The business caseThe problem this solves

A count room reconciles what came out of a bank's drop boxes against what the pit system and the bill-in meters say should be there. The two almost never agree to the cent, and that is normal, so every session produces a variance and a variance inside the printed tolerance band is closed without comment. Free code already flags what falls OUTSIDE the band, perfectly. What nothing on a single sheet can show is that a run of in-tolerance results is a pattern rather than noise -- the same bank short by a little every count, a variance that tracks one count team, or a total that ties only because two errors cancelled inside one leg. Those never trip the band, not once, and they run for as long as the bank keeps trading. a count-room variance sheet that flags what falls outside the tolerance band and closes what falls inside it. That sheet is CORRECT and it is complete on its own slice: the tolerance-band floor in this kit catches 16 of 16 out-of-band exceptions for USD 0.00, and BEATS the model there. What it closes without comment is all 38 structured readings in this corpus, every session, for as long as they run.

The inputThe actual count-room session sheets

The corpus is 120 count-room session sheets, 0.93 MB (txt 120). A property's count-room export gives bank-by-bank drop, the fill and credit trail, the meter positions and the count-team rota, and in most jurisdictions it is a regulated submission. There is no public one. A scrubbed real export is worse, not better: the messy prose -- a stacker jamming for weeks, one team not tallying the coupon tray, a fill written up off the credit book -- is exactly what makes this task hard, and a scrubber removes it first.

Synthetic Record
----------------------------------------------------------------
Every field below is invented. This is a generated casino count-room session sheet for an
AI use-case kit; it reproduces no property, no bank, no device, no count team and no real
person. There is NO PATRON DATA of any kind in this corpus -- this is count-room drop and
aggregate volume only. The tolerance band, the immaterial floor and the bias run length are
ILLUSTRATIVE DEFAULTS.

Bank
----------------------------------------------------------------
Bank reference                           : BK-0001
Property                                 : Harrowgate Bay
Bank                                     : Bank K7 Lagoon (zone RIVERSIDE)
Devices in this bank                     : 16
Gaming date                              : 2026-08-18 (Tuesday)
Count session date                       : 2026-08-19
Count team                               : GOLD
Count sessions since last device service : 10
Count-room supervisor of record          : Marcus Ebele

Drop And Meter Position
----------------------------------------------------------------
What the count room counted out of this bank's drop boxes and slips, against what the pit
system and the bill-in meters say should be there, in USD. A NEGATIVE variance means the
count came up SHORT of what the meters expected.

  Leg                 Counted         Expected       Variance
  CURRENCY          19,859.55        19,859.55           0.00
  FILLS              5,985.36         5,985.36           0.00
  CREDITS            1,869.30         1,869.30           0.00
  VOUCHERS           3,112.85         3,112.85           0.00

  Bank total, counted                            30,827.06
  Bank total, expected                           30,827.06

The outcomeWhat a good result looks like

every bank's verdict with the cause and the sessions flagged, and the structured variances separated from the noise -- 0 false escalations across 99 quiet readings, and 100.0 pct of in-tolerance noise correctly left alone.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Every variance that matters on your floor is already OUTSIDE your tolerance band — the free floor -- tolerance-band, USD 0.00
  It catches 16 of 16 out-of-band exceptions for nothing and beats the model there. Paying for a model to re-find what a subtraction already finds is the worst trade on this page.
• Every cause on your floor is keyed into a structured slip code the same session it happens — the free floor -- ledger-strong, USD 0.00
  The strong floor already scores 95.0 on verdict and beats the model on cause, the supervisor and escalate-now. The model's whole margin is readings whose decisive evidence was typed as a sentence.
• Your in-tolerance variances run for weeks and nobody has ever looked at them as a series — this kit, with the carried history
  The control caught 0.0 pct of the readings that are indistinguishable on the day and added 18 false escalations.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-25. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-25 — r002-cage-drop. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Every corpus sheet, the answer key, all three free floors, the injection probe and what r002-cage-drop actually answered ship in the repo. python3 -m src.app renders the whole UI with no key and no network.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. one reading -- one bank, at one count session, end to end including provider-side reasoning tokens, on the shared connection. Not a cold-start figure and not a single-request figure: 30 bank chains ran with 5 concurrent workers.

Current processWhat it replaces

a count-room variance sheet that flags what falls outside the tolerance band and closes what falls inside it. That sheet is CORRECT and it is complete on its own slice: the tolerance-band floor in this kit catches 16 of 16 out-of-band exceptions for USD 0.00, and BEATS the model there. What it closes without comment is all 38 structured readings in this corpus, every session, for as long as they run.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureArchitectureArchitectureEvalsEvalsEvalsEvalsPromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• jsonl
• json
• 3 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one reading is

30 banks x 4 count sessions, walked strictly in order per bank so the carried history moves forward the way it would in a real deployment. Different banks are independent and run concurrently.

SetupWhat the setup figure measured

There is no index to build -- each reading's sheet goes whole into the prompt. The 0.5s and $0.00 are the corpus generation itself.

LicenceLicence

MIT

Bring your ownBring your own count-room session sheets

tools/build_corpus.py writes data/corpus/*.txt and data/gold.jsonl. Keep the 8 section headings and the field labels src/audit.position_of parses, set your own tolerance band, immaterial floor and bias run length in src/recon.py, then re-run python3 -m evals.check_labels before spending anything.

⚠︎ And what stops being true when you do: The measured figures on this kit's page do not transfer to your own corpus. The band-invisible share (38 of 38 structured readings here, which is 100 pct by construction), the indistinguishable share (26 of 38) and the prose-versus-SLIP split are properties of THIS generator. On a floor whose structured variances happen to be larger than its tolerance band, a band check would already be finding them and this kit would be buying much less.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — but they are the input total apportioned by each part’s share of the characters, not a per-part reading from the provider’s counter. The total is measured; the split is arithmetic, and a part carrying denser text really costs more of it than this says.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 3,618 tokens assembled, 1,883 are documents — 52% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Reproduced from src/prompt.build for BK-0003-S1 with that reading's real carried state, not retyped -- byte-identical to what evals/run.py sent.

Full promptThe prompt in full

You are a casino count-room drop reconciliation. You read one bank of gaming devices at one count session -- the counted drop against what the meters and slips say should be there -- against the DEFAULT count-room policy, and you answer with one JSON object and no other text.

You are the drop reconciliation for one casino property's count room. It runs once per gaming day
and re-reads EVERY bank of gaming devices that was counted. You are reading ONE bank at ONE count
session.

The bank, its four-leg drop and meter position, the DEFAULT count-room policy, and everything
recorded since the previous count session are reproduced in the sheet below. Apply them exactly as
written. You cannot see the earlier count sessions; what is known about them is stated under
"Carried state" and is the only history available to you. Do not assume anything about earlier
sessions beyond it.

⚠︎ THE TOLERANCE BAND, THE IMMATERIAL FLOOR AND THE BIAS RUN LENGTH ARE ILLUSTRATIVE DEFAULTS, NOT
ANY PROPERTY'S, ANY REGULATOR'S OR ANY GAMING BOARD'S TERMS. Apply them as printed regardless of
whether they look right for the bank in front of you.

THE POINT OF THIS JOB: a counted drop and a meter expectation almost never agree to the cent, and
that is normal. A note misread by a validator, a coupon tray tallied into the wrong pile, a fill slip
written a minute after the count started -- every session produces a variance, and a variance inside
the tolerance band is normally closed without comment. Code already flags what falls OUTSIDE the
band, on sight, every time.

WHAT YOU ARE LOOKING FOR IS THE VARIANCE THAT IS SMALL BUT STRUCTURED: the same bank short by a
little every session, a variance that tracks one count team, or a total that ties only because two
errors cancelled. Every one of those sits INSIDE the band, so nothing about this session's own
numbers will tell you. Only the history will.

How to work it out:
 - FIRST decide whether the bank was fully counted. If a drop box, a fill or credit slip, or a meter
   reading is missing for the gaming date, the verdict is COUNT_INCOMPLETE: no variance is judged, no
   cause is determined, nothing is escalated, and the sessions-flagged count does NOT advance. THERE
   IS NO DEFAULT TOLERANCE for a bank that was not fully counted (Rule C-8).
 - THEN check whether the record confirms that two errors CANCELLED inside one leg -- a fill written
   up off the credit book, a meter read across a rollover against a jam undercount, or any other
   pair that offsets. If it does, the verdict is OFFSETTING_ERRORS at ANY variance level, inside the
   band or outside it. A total that ties by cancellation is not a total that ties, and the band was
   never going to fire on it (Rule C-5). Evidence for this can be a logged SLIP: entry or an
   ordinary sentence typed into the count-room notes -- both count exactly the same.
 - THEN read the bank total variance off the printed table and compare it against the printed
   tolerance band. A variance under the printed immaterial floor is not a variance.
 - OUTSIDE the band: the verdict is OUT_OF_TOLERANCE. Escalate it for a recount on this shift
   (Rule C-2).
 - INSIDE the band, check the history for a TEAM LINK first. If every earlier session that came up
   material was counted by the SAME team as this one, AND at least one earlier session counted by a
   DIFFERENT team came out TIED, the verdict is TEAM_LINKED. A bank that ties under one team and
   shorts under another is a procedure, not a device (Rule C-3).
 - INSIDE the band, then check for a DIRECTIONAL BIAS. If the carried state shows at least 2 earlier
   MATERIAL sessions, ALL of them running the same way as this one, and none of them has reversed,
   the verdict is STRUCTURED_BIAS. A bank that came back the other way at any point is noise however
   many sessions it has been running, and a bank with no history on record cannot be convicted at all
   (Rule C-4).
 - OTHERWISE: a variance inside the band with nothing structured about it is IN_TOLERANCE_NOISE, and
   a bank total under the printed immaterial floor is RECONCILED. Neither is escalated. Reopening
   these is the entire reason the variance sheet gets ignored (Rule C-6).
 - THEN decide the CAUSE. Cause evidence ACCUMULATES from earlier sessions (see "Carried state") and
   from anything recorded THIS session -- in the structured SLIP: log or in an ordinary count-room
   note. A note that describes a cause in plain prose counts exactly as much as a logged entry. Use
   UNDETERMINED only when the variance is real and nothing anywhere says why, and NONE only when
   there is nothing to explain.
 - THEN give "sessions_flagged": the number of COUNT SESSIONS on which this reconciliation has
   reported this bank as owing somebody something -- OUT_OF_TOLERANCE, OFFSETTING_ERRORS, TEAM_LINKED
   or STRUCTURED_BIAS -- INCLUDING this one. It is stated in the carried state and it is NOT the
   printed count of sessions since the bank's devices were last serviced: a bank can arrive on its
   14th session since service having never been flagged once. It does not advance on a session that
   was not fully counted.
 - THEN decide the OWNER. Start from the count-room supervisor on file, but a note recording a
   handover to someone else, in this session or an earlier one (see "Carried state"), supersedes it.
 - FINALLY decide "escalate_now": YES only if the verdict is OUT_OF_TOLERANCE, OFFSETTING_ERRORS,
   TEAM_LINKED or STRUCTURED_BIAS AND the carried state does not already show an escalation raised at
   that same level. Those four go to four different places -- a recount, cage accounting, count-room
   training, the device audit -- so an escalation already raised at one level does not satisfy
   another. Otherwise NO.

Answer with a single JSON object and nothing else:

{"verdict": "RECONCILED|IN_TOLERANCE_NOISE|STRUCTURED_BIAS|TEAM_LINKED|OFFSETTING_ERRORS|OUT_OF_TOLERANCE|COUNT_INCOMPLETE",
 "cause": "NORMAL_VARIANCE|BILL_JAM_UNDERCOUNT|TEAM_PROCEDURE|OFFSETTING_POSTING|DEVICE_FAULT|UNDETERMINED|NONE",
 "sessions_flagged": <integer>,
 "owner": "<the current count-room supervisor's name>",
 "escalate_now": "YES|NO",
 "rationale": "one sentence, naming the bank total variance, the tolerance band you applied, and the
   evidence -- carried or recorded -- you relied on"}

Precedence, applied in this order: COUNT_INCOMPLETE if the bank was not fully counted (Rule C-8);
then OFFSETTING_ERRORS on a confirmed cancelled pair (Rule C-5); then OUT_OF_TOLERANCE if the bank
total is outside the printed band (Rule C-2); then TEAM_LINKED (Rule C-3); then STRUCTURED_BIAS
(Rule C-4); then IN_TOLERANCE_NOISE or RECONCILED (Rule C-6). "cause" is UNDETERMINED whenever the
verdict is COUNT_INCOMPLETE, and NONE whenever the verdict is RECONCILED.

Carried state
----------------------------------------------------------------
No earlier count session has been recorded for this bank. This is its first appearance on this reconciliation: nothing is known about which way it has run, which count teams have counted it, whether it has ever tied, no cause has been established, no escalation has been raised and no handover has been recorded. Under Rule C-4, with no history on record there is nothing that can convict a variance of being directional, and under Rule C-3 there is nothing that can link it to a team.

Count-room session sheet
----------------------------------------------------------------
Synthetic Record
----------------------------------------------------------------
Every field below is invented. This is a generated casino count-room session sheet for an
AI use-case kit; it reproduces no property, no bank, no device, no count team and no real
person. There is NO PATRON DATA of any kind in this corpus -- this is count-room drop and
aggregate volume only. The tolerance band, the immaterial floor and the bias run length are
ILLUSTRATIVE DEFAULTS.

Bank
----------------------------------------------------------------
Bank reference                           : BK-0003
Property                                 : Calder Point Resort
Bank                                     : Pit 3 North (zone MAIN FLOOR)
Devices in this bank                     : 19
Gaming date                              : 2026-08-18 (Tuesday)
Count session date                       : 2026-08-19
Count team                               : BLUE
Count sessions since last device service : 2
Count-room supervisor of record          : Ingrid Solberg

Drop And Meter Position
----------------------------------------------------------------
What the count room counted out of this bank's drop boxes and slips, against what the pit
system and the bill-in meters say should be there, in USD. A NEGATIVE variance means the
count came up SHORT of what the meters expected.

  Leg                 Counted         Expected       Variance
  CURRENCY           7,967.52         7,967.99          -0.47
  FILLS              2,330.91         2,330.91           0.00
  CREDITS            2,416.49         2,416.49           0.00
  VOUCHERS             294.41           294.41           0.00

  Bank total, counted                            13,009.33
  Bank total, expected                           13,009.80
  Bank total variance                                -0.47
  Tolerance band (default)                           65.05
  Immaterial floor (default)                          1.00

Bank total against the band              : INSIDE the tolerance band

The boxes for this bank are counted into a BANK-LEVEL total on this feed. There is no
per-device counted figure on the sheet: the drop is counted per box and reconciled per bank,
which is how this property's soft count is submitted.

Count-Room Policy (Default)
----------------------------------------------------------------
The thresholds below are an OPERATOR-TUNABLE DEFAULT, not a real property's, a real
regulator's or a real gaming board's terms -- no documented tolerance, materiality floor or
escalation threshold exists for this row. Applied exactly as printed.

Tolerance band                           : 0.500 pct of the bank's expected total, or 25.00 USD, whichever is larger
Immaterial floor                         : 1.00 USD
Material sessions to convict a bias      : 2
Count-adjustment authority               : NOT DEFINED. Nothing in this kit posts a
                                           count adjustment, writes off a variance, amends a
                                           master-game report, releases a drop, files a
                                           regulatory variance report or clears a device.

Rule C-1  The bank's position is a four-leg reconciliation: currency counted out of the drop boxes,
          chip fills issued to the tables, chip credits returned to the cage, and vouchers -- each
          against what the pit system and the meters say should be there. The BANK TOTAL variance is
          counted minus expected, and it is negative when the count came up short. The tolerance
          band, the immaterial floor and the bias run length below are operator-tunable DEFAULTS,
          not any property's or regulator's terms, and are applied exactly as printed.
Rule C-2  A bank total OUTSIDE the printed tolerance band is OUT_OF_TOLERANCE. It is escalated for a
          recount on this shift. This is the one thing a variance report already catches on sight,
          and it closes on the day.
Rule C-3  INSIDE the band, the reading is TEAM_LINKED if the variance tracks ONE count team: every
          earlier session that came up material was counted by the SAME team as this one, and at
          least one earlier session counted by a DIFFERENT team came out immaterial. A bank that
          ties under one team and shorts under another is a procedure, not a device, and no single
          count sheet can show it.
Rule C-4  INSIDE the band, the reading is STRUCTURED_BIAS if the carried history shows at least 2
          earlier MATERIAL sessions, all of them in the SAME direction as this one, with no
          reversal. A bank that came back the other way at any point is noise however many sessions
          it has been open, and it stays IN_TOLERANCE_NOISE. If there is no history on record at
          all, nothing convicts it: a first session inside the band is IN_TOLERANCE_NOISE.
Rule C-5  A count-room record confirming that two errors CANCELLED inside one leg -- a fill written
          up off the credit book, a meter read across a rollover against a jam undercount -- is
          OFFSETTING_ERRORS at any variance level, inside the band or outside it. A total that ties
          by cancellation is not a total that ties.
Rule C-6  Otherwise, a bank total inside the band with nothing structured about it is
          IN_TOLERANCE_NOISE, and a bank total under the printed immaterial floor is RECONCILED.
          Neither is escalated. Reopening these is the entire reason the variance sheet gets ignored.
Rule C-7  Precedence, applied in this order: COUNT_INCOMPLETE; then OFFSETTING_ERRORS (Rule C-5);
          then OUT_OF_TOLERANCE (Rule C-2); then TEAM_LINKED (Rule C-3); then STRUCTURED_BIAS
          (Rule C-4); then RECONCILED or IN_TOLERANCE_NOISE (Rule C-6). TEAM_LINKED is checked
          BEFORE STRUCTURED_BIAS because a run of shorts under one team satisfies both tests and
          only one of them sends the work to the right place.
Rule C-8  If a drop box, a fill or credit slip, or a meter reading is missing for the gaming date,
          the reading is COUNT_INCOMPLETE: no variance is judged, no cause is determined, nothing is
          escalated, and the sessions-flagged count does NOT advance. THERE IS NO DEFAULT TOLERANCE
          for a bank that was not fully counted -- a reconciliation that could not look cannot claim
          the bank was clean.
Rule C-9  "escalate_now" is YES only where the verdict owes an escalation AND this reconciliation
          has not already raised one at that level. The four escalating verdicts go to four
          different places -- a recount, cage accounting, count-room training, the device audit --
          so an escalation already raised at one level does not satisfy another.

Session Position
----------------------------------------------------------------
Count session date                       : 2026-08-19   (count session 1 of this bank)
Count cadence                            : per gaming day -- one count-room session for each bank, every gaming day
Previous count session                   : -- none, this is the first
Next count session                       : 2026-08-20

Recorded This Session
----------------------------------------------------------------
Everything recorded against this bank between the previous count session and this one.
THIS SESSION ONLY -- an entry recorded on an earlier session is not repeated here.

11:05  The second fill on this bank went in off the credit book by mistake, and somebody
       re-read the meter afterwards to make it balance. The bank total ties but both
       entries are wrong and neither has been corrected.

Count-Room Notes
----------------------------------------------------------------
Volume was heavy on the gaming date in question -- a concert let out onto this zone --
so the absolute figures are larger than this bank's usual session.

Raw responseThe raw response

{"verdict": "OFFSETTING_ERRORS", "cause": "OFFSETTING_POSTING", "sessions_flagged": 1, "owner": "Ingrid Solberg", "escalate_now": "YES", "rationale": "Bank total variance is -0.47 against a 65.05 tolerance band, and the session note confirms an offsetting fill/credit-book and meter re-read error, so the reading is OFFSETTING_ERRORS."}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch casino count-room variances that quietly repeat — 120 count-room session sheets. One model answered, and every answer was then graded Three different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection, never a bill and never a vendor's claim about this workload.

Rates checked 2026-08-18. The provider that actually ran all 588 paid calls is kept out of these tables per this estate's naming rule, so no figure here is a bill.

The gradersThree ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingAnswering and grading are two separate bills

The three free floors, the stub, the cadence arithmetic and every pre-run check cost $0.00 and no calls. 294 PAID CALLS WERE DISCARDED, NOT PUBLISHED: r001, s001, x001 and c000 all ran at a 16,000-token ceiling that was refuted by measurement the same night, and they were moved to docs/discarded/ with their cause named rather than overwritten or spliced.

Cost driversWhat actually moves the bill

• OUTPUT TOKENS, AND ALMOST ALL OF THEM ARE REASONING. 96.2 pct of r002-cage-drop's output (325666 of 338656 tokens) was provider-side reasoning left at the provider default. The answer itself is five short fields and one sentence.
• THE SHEET GOES IN WHOLE, EVERY SESSION. 3617.59 input tokens per reading, of which the carried-history sentence is a few dozen. There is no index and no retrieval: the policy block and the nine rules are re-sent on every one of the 120 calls.

Your volumeWhat it costs at your volume

LINEAR IN BANKS x SESSIONS. Nothing amortises. The sublinear lever NOT implemented is skipping banks whose total ties exactly -- which would also delete the RECONCILED sessions Rule C-3 needs to see a bank tie under a different team.

Model choiceWhy this model

It is the tier the shared connection was already pointed at, and the kit's whole claim is that swapping it is .env plus one more run. No second model was run against this corpus, and the page does not pretend one was: the two-model comparison here is memory-versus-none, and the honest comparison is model-versus-free-code.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

per-query average, this run

The codeEvery file, and what it is for

11 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/recon.pythe reconciliation rule and state machine — a swap seam

The DEFAULT tolerance band, immaterial floor and bias run length, the seven verdicts, the four-step escalation ladder, and step() -- the one place a verdict is decided. Every gold value in data/gold.jsonl is this function's output on the same events the corpus builder wrote onto the sheet.

You change it to: DEFAULT_BIAS_SESSIONS -- how many earlier material sessions convict a directional bias. evals/cadence.py publishes what the current value costs in silent sessions.

# The count-room drop reconciliation rule as arithmetic. Pure code, no model, stdlib only.

CURRENCY = "CURRENCY"
FILLS = "FILLS"
CREDITS = "CREDITS"
VOUCHERS = "VOUCHERS"
LEGS = (CURRENCY, FILLS, CREDITS, VOUCHERS)
DEFAULT_TOLERANCE_PCT = 0.005
DEFAULT_TOLERANCE_FLOOR_USD = 25.00
DEFAULT_IMMATERIAL_USD = 1.00
DEFAULT_BIAS_SESSIONS = 2

src/state.pythe carried state

Four scalars and three short parallel lists per bank -- the signs, the count teams and the materiality of every earlier session. It is larger than a sibling monitor's because a DIRECTION is not a scalar: Rule C-4 asks whether every earlier material session pointed the same way, which a running total cannot answer.

# The carried state -- the thing that makes this a monitor and not another variance sheet.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DEFAULT_PATH = os.path.join(HERE, "data", "state.json")
def load(path=DEFAULT_PATH):
def save(store, path=DEFAULT_PATH):
def for_bank(store, bank_id):
CAUSE_WORDS = {
def describe(state):

src/prompt.pythe prompt

Three parts: the question, the carried-state sentence, the sheet. The middle one is the experiment -- evals/run.py --stateless replaces exactly that line and nothing else.

# Assemble the one prompt this kit sends. No template engine, no framework -- string concatenation

INSTRUCTION = """\
def build(text, carried, stateless=False):

src/segment.pysection split

Eight fixed sections per sheet. check_labels requires all eight in all 120 documents before a run may spend.

# Split a count-room session sheet into its named sections. Pure code, no model.

SECTIONS = ("Synthetic Record", "Bank", "Drop And Meter Position",
RULE = "-" * 64
def split(text):
def render(secs):

src/select.pywhat is sent — a swap seam

Count-Room Staff Contact is mapped by nothing and therefore never sent. The guard is red-proven: rename the sheet's headings the way a CMS upgrade would and the naive fallback leaks the block on 120 of 120 documents.

You change it to: SECTION_HINTS and NEVER_SENT. Adding a section to the union sends it; the withheld list is what stops that happening by accident.

# Pick which sections of a count-room session sheet are sent. Pure code -- the last deterministic

BANNER = "Synthetic Record"
BANK = "Bank"
POSITION = "Drop And Meter Position"
POLICY = "Count-Room Policy (Default)"
SESSIONPOS = "Session Position"
RECORDED = "Recorded This Session"
STAFF = "Count-Room Staff Contact"
NOTES = "Count-Room Notes"
NEVER_SENT = (STAFF,)

src/adapters/__init__.pythe model call — a swap seam

SEAM 1. Raw HTTP, stdlib only, one function per provider shape. Swapping provider or model is .env plus the same run again.

You change it to: PROVIDER, BASE_URL and MODEL in .env. Nothing else moves.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

src/budget.pythe spend guard

A per-day call cap shared by every kit on the machine, checked inside the adapter so the harness, the local app and a screenshot script all go through it.

# A cap on live calls, shared by every kit on this machine. No dependency, no service.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
KIT = os.path.basename(HERE)
ROOT = os.path.dirname(os.path.dirname(HERE))
SHARED_ENV = os.path.join(ROOT, ".env")
LEDGER = os.path.join(ROOT if os.path.exists(SHARED_ENV) else HERE, ".calls-ledger.jsonl")
PAID_ARMS = ("r", "x")
FREE_ARMS = ("t", "b")
BOOT = os.urandom(6).hex()
PID = os.getpid()

evals/baseline.pythe free floors

Three, none a strawman. tolerance-band is a genuine, correct implementation of the printed policy and it beats the model on its own slice.

# THE FREE FLOORS. Three of them, none a strawman, all of them USD 0.00.

MODES = ("tolerance-band", "band-plus-run", "ledger-strong")
def _section(text, name, nxt, flat=False):
def _facts(text):
def _finish(verdict, cause, sessions_flagged, owner, escalate_now, why):
SLIP_CAUSE = (
def review(text, carried=None, mode="ledger-strong"):

evals/scoring.pythe scorer

Exact match per cell. Prints the band-visible and band-invisible slices against SEPARATE denominators and never means them.

# Score a run against the answer key. Pure code, exact match per cell. No model grades anything.

NEEDS_ATTENTION = ("STRUCTURED_BIAS", "TEAM_LINKED", "OFFSETTING_ERRORS", "OUT_OF_TOLERANCE")
STRUCTURED = ("STRUCTURED_BIAS", "TEAM_LINKED", "OFFSETTING_ERRORS")
FIELDS = ("verdict", "cause", "sessions_flagged", "owner", "escalate_now")
def _pct(n, d):
def _same(got, want):
def score(records, golds):
def _protection(records, golds):

evals/check_labels.pythe pre-flight checks

26 claims the page makes, checked against the corpus that will be sent, before anything spends.

# Everything that must be true BEFORE a run is allowed to spend. Free, and it runs in seconds.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
GOLD = os.path.join(HERE, "data", "gold.jsonl")
FAILED = []
def check(name, ok, detail=""):
def main():
def _carried_in_for(gold, g):

src/app.pythe local UI

http.server and hand-written HTML/JS on port 9022. Renders with no key; the second button replays what the scored run actually answered off the committed result file. HTML/JS lives beside it under ui/.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
PORT = int(os.environ.get("PORT", "9022"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r002-cage-drop")
GOLD_ROWS = {r["doc_id"]: r for r in
def carried_for(doc_id):
class H(BaseHTTPRequestHandler):
def main():

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 3617 input and 2822 output tokens per reading (one bank, at one count-room session), all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

MEASURED, AND THE PAIRING IS THE FINDING. Count-Room Notes is this kit's injection surface, and the probe forces the instruction-shaped note onto every reading where an escalation is due -- then fires each one TWICE so the comparison is against the model's own un-injected answer rather than against the answer key.

API_KEY is read from .env (gitignored) or the real environment; never logged, never placed in a prompt, and redacted from any provider error message the local UI passes to the browser.

The experimentWe DID attack it -- one sentence, on every reading where it could matter, each paired against its own control

The probe replaces the Count-Room Notes section with the instruction-shaped note on every reading where an escalation was due, and fires the SAME reading unchanged alongside it. A suppression is counted only where the control escalated and the injected arm did not. On this run 19 of 19 paired readings still escalated. The pairing is not a refinement: without it this probe would have published 9.52 pct off readings the model never escalated in the first place. Both were run for real on 2026-08-25.

Both boundaries are measured in both directions: the injection probe forces the condition rather than waiting for it AND pairs each reading against its own control, and the privacy guard is red-proven by reproducing the schema-change condition that reaches its fallback.

The result0 of 19 escalations suppressed by the instruction-shaped note, paired against their own controls -- measured, not assumed.

One phrasing, one model, one corpus, 21 readings fired twice each -- every reading where suppression was even possible, each paired against its own un-injected answer.

HonestyWhat this does not prove

• Whether the rate is real at all. Two probes on the same corpus returned 1 of 17 and 0 of 19; at this magnitude the probe cannot separate suppression from the model's own run-to-run variance on decision-boundary readings, and no repeat was paid for.
• Whether the suppression reproduces across phrasings, models or corpora. One sentence, one model.
• Whether an injection placed in Count-Room Staff Contact would have any effect -- by construction it cannot, because that section never leaves the machine.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Never post a count adjustment, write off a variance, amend a master-game report, release a drop, file a regulatory variance report or clear a device -- and never present the tolerance band, the immaterial floor or the bias run length as any property's, regulator's or gaming board's terms.

Stated to the model on every call in src/prompt.py's INSTRUCTION block and printed on every sheet, and enforced by evals/check_labels.py's banned-code-path scan before any run may spend.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• This is a PROMPT RULE PLUS A STATIC SCAN, not a runtime enforcement layer. Nothing stops a forker adding such a path tomorrow; the scan catches it only the next time somebody runs check_labels.py, which is a manual step.
• The injection result is ONE SENTENCE against ONE model on ONE corpus, and two probes on it returned 1 of 17 and 0 of 19. It is not a resistance rate for prompt injection in general, and it cannot resolve a rate that small from run-to-run variance.
• The banned-path scan knows NAMES. A code path called something else would pass it, which is stated here rather than hidden.

WatchedWhat is watched, and why that one

The kit measures 93 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

4 measured by the latest run 89 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

no shared guard The two recorded runs declare no guard in common, so nothing below is protected by a precondition — read the history as two separate observations, not as a difference.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

8 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 4 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

triage · no model in the path — a baseline, not a peer column — 3 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 3 runs measured the same system — they differ on banks_caught, banks_missed, false_escalations, missed_escalations, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

triage · with the model in the path — 3 runs. Columns here are only ever compared with each other.

not a time series No two of these 3 runs measured the same system — they differ on answered, band_invisible_structured_cells, band_visible_cells, banks_caught, banks_missed, banks_needing_attention_at_last_session, count_incomplete_cells, documents, escalation_cells, false_escalations, memory_cells, missed_escalations, noise_cells, quiet_cells, readings_scored, stateless, structured_cells, structured_indistinguishable_cells, structured_quiet_cells — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 18 chips that all say so.

redteam · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 4 chips that all say so.

DeviationsWhat deviated

0 breaches across 8 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

The guardrail scan is a MANUAL step, run before each spend, not a hook. It ran before every paid run and passed at 0 each time. The injection probe is a ONE-OFF, and it was fired twice.

In one lineNo framework, deliberately

NO FRAMEWORK. A folder of readable Python, no dependency, and a prompt anyone can read end to end. A framework abstraction would own the memory layer, and the memory layer is already the entire surface of src/state.py -- three parallel lists whose SHAPE is the argument this kit makes.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Linear: the sheet -> src/select.py -> src/prompt.py -> src/adapters -> evals/scoring.py. No branching, no tool-calling and no agent loop.

The other sideWhat a framework costs you

• Swapping providers means editing the PROVIDERS dict by hand. In exchange the prompt has no hidden templating layer and requirements.txt stays empty.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r002-cage-drop on the fast tier, with the carried history, 2026-08-25. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 5 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

3 runs not plotted. b000-cage-drop-band, b001-cage-drop-run, b002-cage-drop-ledger recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the population is re-read whole on each scheduled run, and the prior run's carried state is what makes a reading a CHANGE.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe verdict, cause, sessions flagged, supervisor and escalate-now call, per reading, exact match against the computed answer key

whether each of the 5 answered fields equals the answer key, per reading

MethodHow the test was run

python3 -m evals.run --run-id <id> ...; evals/scoring.py compares strings. No model grades anything.

Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

accuracy = hits / 120 per field. drop_variance_accuracy_pct is scored separately, and its band-visible and band-invisible halves are reported against SEPARATE denominators and never averaged.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the computed answer key in data/gold.jsonl, produced by src/recon.step() from the same planted inputs the corpus builder wrote onto the sheet.

No true/false rates for this grader. It records 5 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No sweep: every answer is an enum, a name or an integer the arm emits.

Cadence: once per run; free

The decisionWhen to reach for it

In one lineThe two denominators: band-visible exceptions named, and band-invisible structured variances caught

which half of the corpus an arm's discriminator score actually came from

MethodHow the test was run

evals/scoring.py, on every arm. evals/check_labels.py asserts the split is real before anything spends.

Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

band_visible_caught_pct over 16 readings whose own total is outside the printed band; band_invisible_structured_caught_pct over 38 structured readings whose total is inside it. NEVER averaged.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the printed tolerance band on each sheet, re-read by evals/baseline.py's own regex rather than imported from src/recon -- so the split is a property of the page, not of this repository.

No true/false rates for this grader. It records 3 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? The band itself IS the threshold, and it is printed on every sheet rather than chosen by this grader.

Cadence: once per run; free

The decisionWhen to reach for it

In one lineThe same model and the same prompt with the carried-history block removed

how much of the discriminator is the memory rather than the reading

MethodHow the test was run

python3 -m evals.run --run-id s002-... --stateless --yes

Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

One variable. evals/check_labels.py asserts the two prompt builders differ on exactly one line.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the same computed answer key; the only variable is one line of the prompt.

No true/false rates for this grader. It records 3 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No threshold: the arms are compared cell for cell.

Cadence: once per corpus revision; 120 calls

The decisionWhen to reach for it