The business caseThe problem this solves

A franchisee declares gross revenue every month and the franchisor charges a royalty on it. Recomputing that royalty is arithmetic and a spreadsheet does it perfectly, so the statement worth finding is the tidy one: correctly footed, filed on time, plausible every month, and short. Nobody audits every property every period, so the only evidence available is what the franchisor already holds -- how many room nights the property sold, what its own brand channel captured, and what it declared last month. a royalty audit rota. Nobody audits every property every period, so a franchisor's real control is a spreadsheet that recomputes the royalty from the declared figure -- which is right about the arithmetic on every one of these 120 statements and finds 0 of the 28 that are short.

The inputThe actual royalty-run extracts

The corpus is 120 royalty-run extracts, 0.72 MB (txt 120). A hotel brand's per-property declared revenue, occupancy and royalty file is the single most commercially sensitive table a franchisor holds, and there is no public one. A scrubbed export is worse rather than better: scrubbing removes the franchisee's own note, which is exactly the layer this kit measures.

Synthetic Record
----------------------------------------------------------------
This file is SYNTHETIC. It was generated by tools/build_corpus.py for the AI Foundry use-case kit
`franchise-royalty`. No real franchisor, franchisee, property, person, agreement or revenue figure
appears anywhere in it, and the royalty schedule reproduced below is an illustrative default rather
than any real brand's terms. It is safe to read, to copy and to publish.

Property
----------------------------------------------------------------
Property reference                      : FR-0001
Property                                : Harbour Court Inn, Bristol
Rooms in inventory                      : 118
Franchise agreement                     : FA-2015-0001
Franchise business consultant of record : Nadia Okonjo
Royalty period                          : 2025-P10 (2025-10-01 to 2025-10-31)
Royalty run                             : 2025-11-05  (royalty run 1 of this property)
Previous royalty run                    : -- none
Next royalty run                        : 2025-12-05

Declared Statement
----------------------------------------------------------------
Filed by the franchisee on 2025-11-02, reproduced exactly as filed.

  Rooms revenue, gross                                397,282.72
  Food and beverage revenue, gross                     59,195.13
  Other operating revenue, gross                       24,234.25
  Gross revenue, all categories                       480,712.10

  Declared out of the royalty base
    Third-party concession F&B                         36,878.57
    Guest-recovered utility surcharge                   4,386.40
  Royalty-bearing revenue declared                    439,447.13

  Royalty rate declared                                    5.50 %

The outcomeWhat a good result looks like

every property's verdict with the governing signal and the runs-open count, and the credible statements separated from the short ones -- 1 false queries across 97 quiet readings.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Your franchisees always file against a cause code — the free floor -- schedule-strong, $0.00
  It reaches 92.5 pct on verdict and 100.0 pct on the coded-cause readings, catches 100.0 pct of the under-reported readings and 100.0 pct of the exclusion abuse, all for nothing.
• Franchisees explain shortfalls in their own words, in a free-text note — the model, with the carried position
  99.17 pct on verdict against the floor's 92.5, and 88.89 pct on the prose-cause readings against 0.0.
• You want the reading but cannot carry state between royalty runs — the model, and accept a named loss
  The stateless control matches on the discriminator (100.0 pct) but scores 0.0 pct on self-corrected recall and raises 18 false queries on 97 quiet readings.

And where nothing here is good enough:

• All you have is the declared series and a variance report — neither, and the number says why
  The variance floor caught 0.0 pct of the 15 internally-consistent under-reports and raised 28 false under-report calls on 92 clean readings. A band on a series cannot see a property that declares inside its own trailing mean while its occupancy moves.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-25. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-25 — r002-franchise-royalty. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Every corpus extract, the answer key, all three free floors, the paired injection probe, the cadence probe and what r002-franchise-royalty actually answered ship in the repo. python3 -m evals.check_labels runs 14 checks and --self-test red-proves four of them, all with no key and no network.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. one reading -- one property, at one monthly royalty run, end to end including provider-side reasoning tokens, on the shared connection. Not a cold-start figure and not a per-property SLA: 120 readings ran in 449.6 wall seconds because different properties run concurrently while a single property's six periods are strictly serial.

Current processWhat it replaces

a royalty audit rota. Nobody audits every property every period, so a franchisor's real control is a spreadsheet that recomputes the royalty from the declared figure -- which is right about the arithmetic on every one of these 120 statements and finds 0 of the 28 that are short.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureArchitectureArchitectureEvalsEvalsEvalsEvalsPromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• jsonl
• json
• 3 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one reading is

20 properties x 6 monthly royalty runs, walked strictly in order per property so the carried state moves forward the way it would in a real deployment. Different properties run concurrently; a single property's six periods never do.

SetupWhat the setup figure measured

There is no index to build -- each reading's extract goes whole into the prompt. The figures are the corpus generation itself.

LicenceLicence

MIT

Bring your ownBring your own royalty-run extracts

tools/build_corpus.py writes data/corpus/*.txt and data/gold.jsonl. Keep the 9 section headings and the field labels src/figures.py parses. Set your own rate, permitted exclusions, tolerance, signal precedence and approved-cause list in src/schedule.py FIRST -- every one of them is an illustrative default and none is any real franchisor's terms.

⚠︎ And what stops being true when you do: The measured figures on this page do not transfer to your own corpus. The band-invisible share (15 of 28 under-reported readings) and the prose-versus-code evidence split (9 readings) are properties of this generator's declared distribution, not facts about hotel franchising. Re-run the evals on your own data; that is what the harness is for.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 3,371 tokens assembled, 1,703 are instructions — 51% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Assembled by src/prompt.build for FR-0011-P6 and reproduced byte for byte. The stateless control sends this prompt with the carried-state block replaced by one sentence and nothing else; evals/check_labels.py check 11 asserts the two builders agree on every other byte.

Full promptThe prompt in full

You are the monthly royalty reconciliation for one hotel brand's franchised estate. You read one property's self-declared revenue statement for one period, against the corroborating signals the franchisor already holds and the DEFAULT royalty schedule, and you answer with one JSON object and no other text.

You are the monthly royalty reconciliation for one hotel brand's franchised estate. It runs five
days after each period closes and re-reads EVERY open property. You are reading ONE property at ONE
royalty run.

The franchisee's declared statement, the royalty schedule, the corroborating signals the franchisor
already holds, this property's own prior declared periods, and everything recorded in the period are
reproduced in the extract below. Apply the schedule exactly as printed. You cannot see the earlier
royalty runs; what is known about them is stated under "Carried state" and is the only history
available to you. Do not assume anything about earlier runs beyond it.

⚠︎ THE RATE, THE PERMITTED EXCLUSIONS, THE TOLERANCE, THE MATERIALITY FLOOR AND THE APPROVED-CAUSE
LIST ARE ILLUSTRATIVE DEFAULTS, NOT ANY REAL FRANCHISOR'S TERMS. Apply them as printed regardless of
whether they look right for the property in front of you.

THE POINT OF THIS JOB: the franchisee files the revenue figure and the franchisor charges a royalty
on it. Recomputing that royalty from the declared figure is arithmetic and a spreadsheet does it
perfectly, so it is NOT what you are for. What you are deciding is whether the DECLARED FIGURE
ITSELF is credible against the evidence the franchisor already holds. The interesting statement is
the one that is tidy: correctly footed, plausible, filed on time, and short.

⚠︎ EVERY TEST BELOW IS A COMPARISON OF FIGURES ALREADY PRINTED ON THE PAGE, PLUS AT MOST ONE
SUBTRACTION. Do not compute percentages of your own -- the 4.0 % tolerance is printed in money, per
signal, in the "Tolerance in money" column.

How to work it out:
 - FIRST find the GOVERNING SIGNAL. It is the first signal in the schedule's printed precedence
   whose coverage is 'complete'. A signal marked 'partial' or 'absent' may NEVER govern, however
   suggestive its figure looks. If NO signal is 'complete', the verdict is SIGNALS_INCOMPLETE: no
   gap is judged, the basis is NONE, no query is raised, and the periods-open count does NOT
   advance. There is no default signal for a feed that never arrived (Rule F-8).
 - THEN check that the statement FOOTS, in four places: the three gross categories must sum to the
   declared gross; gross less the declared exclusions must equal the declared royalty-bearing
   revenue; the declared rate must be the schedule's rate; and the royalty due must be that rate
   applied to that base. Outside the printed footing tolerance on ANY of the four, the verdict is
   ARITHMETIC_ERROR and you stop there -- a statement that does not add up is returned before its
   credibility is argued about (Rule F-1).
 - THEN read every line the franchisee declared OUT of the royalty base against the schedule's list
   of permitted exclusions. A line the schedule does not carry, or a permitted one taken beyond its
   printed cap, is EXCLUSION_ABUSE -- whatever the line is called, and EVEN IF THE GROSS REVENUE IS
   DECLARED PERFECTLY. The gross is not the base (Rule F-2).
 - THEN, if the statement carries a prior-period catch-up adjustment AND the carried state shows a
   gap already open against this property, the verdict is SELF_CORRECTED. On the page this looks
   like revenue ABOVE what the signals imply, which is the shape an anomaly check flags; only the
   carried state tells the two apart (Rule F-7).
 - THEN take the GOVERNING signal's row and read its printed "Gap vs declared". A gap that is not
   larger than BOTH that row's printed "Tolerance in money" AND the schedule's materiality floor is
   not a gap at all: the verdict is CONSISTENT (Rule F-5).
 - IF there is a gap, look for an APPROVED CAUSE dated inside the period. A cause may be entered
   against a code in the event log, or set out by the franchisee in their own words in the note --
   the two count exactly the same, PROVIDED the cause is one the schedule approves. A reason that is
   not on the approved list excuses nothing, however reasonable it is.
 - AN APPROVED CAUSE EXCUSES ONLY UP TO ITS OWN PRINTED IMPACT, AND CAUSES ACCUMULATE. Subtract the
   impacts from the gap. If what is left is not larger than both printed thresholds, the verdict is
   EXPLAINED_MOVE. If what is left is STILL larger than both, the verdict is UNDERREPORTED -- an
   approved cause too small to account for the gap does not excuse the rest of it (Rules F-4, F-3).
 - A gap past both thresholds with nothing approved covering it is UNDERREPORTED (Rule F-3).
 - THEN give "basis": the name of the governing signal you used, or NONE.
 - THEN give "periods_open": the number of royalty runs on which this audit has reported a gap open
   against this property, INCLUDING this one. It is stated in the carried state. It advances on
   UNDERREPORTED and EXCLUSION_ABUSE and HOLDS its value on every other verdict -- it never resets,
   because a corrected shortfall still happened (Rule F-9).
 - THEN decide the OWNER. Start from the franchise business consultant of record, but a handover
   recorded to somebody else, in this period or an earlier one (see "Carried state"), supersedes it.
 - FINALLY decide "raise_query": YES only if the verdict is UNDERREPORTED, EXCLUSION_ABUSE or
   ARITHMETIC_ERROR AND the carried state does not already show a query raised at that same level.
   Those three go to three different desks, so a query already raised at one level does not satisfy
   another. Otherwise NO (Rule F-10).

Answer with a single JSON object and nothing else:

{"verdict": "CONSISTENT|EXPLAINED_MOVE|UNDERREPORTED|EXCLUSION_ABUSE|ARITHMETIC_ERROR|SELF_CORRECTED|SIGNALS_INCOMPLETE",
 "basis": "OCCUPANCY|CHANNEL|SUPPLY|PRIOR_PERIOD|NONE",
 "periods_open": <integer>,
 "owner": "<the current franchise business consultant's name>",
 "raise_query": "YES|NO",
 "rationale": "one sentence, naming the governing signal, the printed gap, anything you netted off
   it and why, and the evidence -- carried or recorded -- you relied on"}

Precedence, applied in this order: SIGNALS_INCOMPLETE if no signal is complete (Rule F-8); then
ARITHMETIC_ERROR if the statement does not foot (Rule F-1); then EXCLUSION_ABUSE (Rule F-2); then
SELF_CORRECTED (Rule F-7); then CONSISTENT if the gap is inside both printed thresholds (Rule F-5);
then EXPLAINED_MOVE if approved causes cover it (Rule F-4); otherwise UNDERREPORTED (Rule F-3).
"basis" is NONE whenever the verdict is SIGNALS_INCOMPLETE and is the governing signal's name in
every other case, including ARITHMETIC_ERROR and EXCLUSION_ABUSE.

Carried state
----------------------------------------------------------------
A credibility gap of 61,488.60 GBP is open against this property, measured against the OCCUPANCY signal at the previous royalty run. The periods-open count carried in is 1. A query has ALREADY been raised at UNDERREPORTED (brand revenue audit). A query already raised at one level does not satisfy another. The franchise business consultant carried in from earlier runs is Nadia Okonjo. The previous royalty run returned CONSISTENT.

Royalty-run extract
----------------------------------------------------------------
Synthetic Record
----------------------------------------------------------------
This file is SYNTHETIC. It was generated by tools/build_corpus.py for the AI Foundry use-case kit
`franchise-royalty`. No real franchisor, franchisee, property, person, agreement or revenue figure
appears anywhere in it, and the royalty schedule reproduced below is an illustrative default rather
than any real brand's terms. It is safe to read, to copy and to publish.

Property
----------------------------------------------------------------
Property reference                      : FR-0011
Property                                : Portland Wharf Hotel, Plymouth
Rooms in inventory                      : 156
Franchise agreement                     : FA-2016-0011
Franchise business consultant of record : Nadia Okonjo
Royalty period                          : 2026-P03 (2026-03-01 to 2026-03-31)
Royalty run                             : 2026-04-05  (royalty run 6 of this property)
Previous royalty run                    : 2026-03-05
Next royalty run                        : -- none

Declared Statement
----------------------------------------------------------------
Filed by the franchisee on 2026-04-02, reproduced exactly as filed.

  Rooms revenue, gross                                297,695.81
  Food and beverage revenue, gross                     47,035.94
  Other operating revenue, gross                       21,731.79
  Gross revenue, all categories                       366,463.54

  Declared out of the royalty base
    Third-party concession F&B                         40,592.02
    Guest-recovered utility surcharge                   3,803.06
  Royalty-bearing revenue declared                    322,068.46

  Royalty rate declared                                    5.50 %
  Royalty due, as declared                             17,713.77
  Prior-period catch-up adjustment included              -- none

Royalty Schedule (Default)
----------------------------------------------------------------
Royalty rate on royalty-bearing revenue      : 5.50 %
Royalty-bearing categories                   : ROOMS, FB, OTHER

Exclusions this schedule permits, and their caps:
  Guest-recovered utility surcharge          against OTHER  up to 40 % of gross OTHER
  Statutory occupancy tax collected          against ROOMS  uncapped
  Third-party concession F&B                 against FB     up to 100 % of gross FB
  Gift certificates not yet redeemed         against OTHER  up to 25 % of gross OTHER
  Any other exclusion is NOT permitted by this schedule, whatever it is called.

Credibility tolerance                        : 4.0 % AND 2,500.00 GBP
  A gap smaller than EITHER of those is not a gap. The 4.0 % is printed in
  money, per signal, in the 'tolerance in money' column of the signal table, so
  every test on this page is a comparison of two figures already printed on it.
Statement footing tolerance                  : 0.50 GBP

Governing corroborating signal, in this order: OCCUPANCY > CHANNEL > SUPPLY > PRIOR_PERIOD
  The governing signal is the FIRST in that order whose coverage is 'complete'.
  A signal whose coverage is 'partial' or 'absent' may NEVER govern.
  If no signal is 'complete', nothing about this statement's credibility is
  judged at all.

Approved causes -- and each excuses only up to its OWN printed impact:
  ROOMS_OUT_OF_SERVICE     rooms removed from saleable inventory for planned refurbishment
  FLOOR_CLOSURE            a wing or floor closed for the period on brand instruction
  BRAND_TRANSITION         property mid-conversion, brand channel revenue routed elsewhere
  FORCE_MAJEURE_CLOSURE    a declared market event closing the property
  GROUP_CANCELLATION       a contracted group cancelled inside the period
  A cause described in the franchisee's own words counts exactly as much as one
  entered against a code, PROVIDED it is one of the causes above. A reason that
  is not on that list excuses nothing, however reasonable it is.

Corroborating Signals
----------------------------------------------------------------
What the franchisor already holds for this property and period. The franchisee files
none of it.

  Occupancy feed (brand property-management extract)   coverage: complete
    Room nights available                                  4836
    Room nights sold                                       3422   (70.8 %)
    Rate applied -- this property's OWN prior-period achieved rate,
    taken from its last declaration                       96.13
  Brand booking channel (central reservations)         coverage: complete
    Bookings captured through brand channels               2155
  Supply orders (linen and amenity shipments)          coverage: partial
  Prior declared periods on file                       coverage: complete
    Periods on file and judged                                5

  Implied royalty-bearing revenue, by signal
    Signal          Coverage    Implied revenue    Gap vs declared   Tolerance in money
    OCCUPANCY       complete           355,888.89          33,820.43          14,235.56
    CHANNEL         complete           350,981.92          28,913.46          14,039.28
    SUPPLY          partial      -- not available
    PRIOR_PERIOD    complete           390,675.92          68,607.46          15,627.04

Prior Periods
----------------------------------------------------------------
  Period       Declared royalty-bearing   Occupancy   Share of gross
  2025-P10          433,083.26       67.6 %         87.9 %
  2025-P11          419,287.01       72.9 %         87.9 %
  2025-P12          444,300.28       74.8 %         87.9 %
  2026-P01          378,993.50       74.1 %         87.9 %
  2026-P02          320,122.87       70.5 %         87.9 %

  Trailing mean of declared royalty-bearing revenue        390,675.92

Events In This Period
----------------------------------------------------------------
  No events were recorded against this property in this period.

Franchisee Note
----------------------------------------------------------------
  The east wing was closed for the period from the 2026-03-03 on brand instruction and what was left went out discounted. That is 33,820.43 against last period's achieved rate.

Raw responseThe raw response

{"verdict": "EXPLAINED_MOVE", "basis": "OCCUPANCY", "periods_open": 1, "owner": "Nadia Okonjo", "raise_query": "NO", "rationale": "The governing OCCUPANCY signal showed a printed gap of 33,820.43 GBP, which the franchisee's note fully attributes to a brand-instructed east-wing closure from 2026-03-03, an approved FLOOR_CLOSURE cause, leaving nothing beyond the printed thresholds."}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Check a hotel franchisee's monthly royalty statement — 120 royalty-run extracts. One model answered, and every answer was then graded one way — by == against a fixed gold value, which is a measurement too and only as good as the gold behind it.

The rulerHow it was graded

Exact string match per cell against a computed key. No LLM-as-judge anywhere in this kit, so no judge to calibrate and no judge cost.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection, never a bill and never a vendor's claim about this workload.

Rates checked 2026-08-18. The provider that actually ran all 328 calls is kept out of these tables per this estate's naming rule, so no figure here is a bill.

Grading unitWhat the grading figure prices

The grader is pure Python over the committed result files and costs nothing to re-run; python3 -m evals.run --floor schedule-strong re-scores the strongest free arm end to end with no key.

The gradersOne way to grade, and why it is the only one

The model does not lose a single column to any floor on this corpus. The floor's own numbers are what carry the information: schedule-strong reaches 92.5 pct on verdict and 0.0 pct on the prose-cause readings, and variance-vs-prior reaches 0.0 pct on the band-invisible ones.

The row opens its own page: how the test was run, one real row, the formula with every iteration of it, and the analysis.

Choosing oneWhen to reach for the model at all

The comparison is only useful if it ends in a choice, and on this kit the choice is not which grader — there is one, and it is ==. It is whether to call a model for this field at all, and the answer is conditional.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingAnswering and grading are two separate bills

The three free floors, the stub, the cadence probe and every pre-run check cost $0.00 and no calls. NO REPLY WAS TRUNCATED IN ANY ARM, but three runs WERE discarded: both scored arms and the injection probe were first taken at a 16,000 ceiling and re-taken at 32,000 after a sibling kit's measurement showed 16,000 was refuted rather than merely unproven. That is 310 calls; the $1.7536 prices the two scored arms on the same projection card, and the discarded probe's 70 calls are unpriced for the same reason the live one is -- the probe records no token totals. The re-take was vindicated: the 32,000 run drew 17,166 output tokens on one reading, above the old cap.

Cost driversWhat actually moves the bill

• OUTPUT TOKENS, AND ALMOST ALL OF THEM ARE REASONING. 94.8 pct of r002-franchise-royalty's output (253216 of 267093 tokens) was provider-side reasoning left at the default. The answer is five short fields and a sentence; the bill is the thinking in front of it.
• THE PROMPT IS MOSTLY FIXED TEXT. 3122.37 input tokens per reading, and the schedule block, the rule text and the instruction are identical on every call -- so a provider with prompt caching would price this workload very differently, and that was not measured.

Your volumeWhat it costs at your volume

LINEAR IN PROPERTIES x PERIODS. Nothing amortises. The sublinear lever NOT implemented is skipping properties inside tolerance on every complete signal -- which on this corpus would skip 92 of 120 readings and would also skip the 7 exclusion-abuse readings, whose gross revenue is declared perfectly.

Model choiceWhy this model

It is the tier the shared connection was already pointed at, and the kit's whole claim is that swapping it is .env plus one more run. No second model was run against this corpus; every other row in the cost table is either a free floor or a projection onto a published card, and both are labelled as such.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

per-query average, this run

The codeEvery file, and what it is for

14 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Seven of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/schedule.pythe royalty schedule, as data — a swap seam

The rate, the permitted exclusions and their caps, the tolerance and materiality floor, the signal precedence and the approved-cause list. One copy; changing it moves the key, all three floors, the prompt and the UI together.

You change it to: APPROVED_CAUSES -- what a franchisee is allowed to explain a shortfall with.

# The royalty schedule and the approved-cause list, as data. No model, no arithmetic.

ROYALTY_RATE_PCT = 5.50
ROYALTY_BEARING = ("ROOMS", "FB", "OTHER")
PERMITTED_EXCLUSIONS = {
NOT_PERMITTED_EXAMPLES = (
TOLERANCE_PCT = 4.0
MATERIALITY_GBP = 2500.00
FOOTING_TOLERANCE_GBP = 0.50
SIGNAL_PRECEDENCE = ("OCCUPANCY", "CHANNEL", "SUPPLY", "PRIOR_PERIOD")
COVERAGE_COMPLETE = "complete"

src/royalty.pythe rule and state machine

Rules F-1..F-10 as arithmetic and step(), which decides the next carried state from the answer key's inputs, never from a model reply. It is also the answer key.

# The royalty-reconciliation rule as arithmetic. Pure code, no model, standard library only.

CONSISTENT = "CONSISTENT"
EXPLAINED_MOVE = "EXPLAINED_MOVE"
UNDERREPORTED = "UNDERREPORTED"
EXCLUSION_ABUSE = "EXCLUSION_ABUSE"
ARITHMETIC_ERROR = "ARITHMETIC_ERROR"
SELF_CORRECTED = "SELF_CORRECTED"
SIGNALS_INCOMPLETE = "SIGNALS_INCOMPLETE"
VERDICTS = (CONSISTENT, EXPLAINED_MOVE, UNDERREPORTED, EXCLUSION_ABUSE, ARITHMETIC_ERROR,
QUERY_VERDICTS = SCH.QUERY_LEVELS

src/figures.pythe printed-figure parser

The statement, the exclusion lines and the four-signal table read exactly, in code. The model is never asked to divide.

# The printed numbers, read off the page in pure code. No model is ever asked to do arithmetic

DATE = r"\d{4}-\d{2}-\d{2}"
MONEY = r"-?[\d,]+\.\d\d"
def _m(text, pat, cast=str, flags=re.M):
def _money(text, pat):
def header(text):
def statement(text):
def signals(text):
def prior_periods(text):
def events_none(text):

src/notes.pythe note reader

coded() is what a regex over the event log can see; stated() is what the written rule says. The distance between the two is the only thing a model is paid to close here.

# Read a franchisee's note as a CAUSE CLAIM. Pure code -- and deliberately only as good as a

DATE = r"\d{4}-\d{2}-\d{2}"
MONEY = r"-?[\d,]+\.\d\d"
PROSE = {
def _section(text, name):
def period_range(text):
def _money(s):
def coded(text):
def stated(text):
def in_period_only(claims):

src/state.pythe carried state

The open gap, the runs-open count and which query levels are already raised. Three of the seven verdicts cannot be reached from one period's page.

# What this audit carries from one royalty run to the next, and the sentences it hands the model.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DEFAULT_PATH = os.path.join(HERE, "data", "state.json")
def load(path=DEFAULT_PATH):
def save(store, path=DEFAULT_PATH):
def for_property(store, property_id):
def describe(state):

src/segment.pythe section splitter

Splits the page into its 9 named sections; asserted across all 120 documents.

# Split a royalty-run extract into its named sections. Pure code, no model.

SECTIONS = ("Synthetic Record", "Property", "Declared Statement", "Royalty Schedule (Default)",
RULE = "-" * 64
def split(text):
def render(secs):

src/select.pythe send filter — a swap seam

Property Contact is mapped by no field and therefore never sent.

You change it to: SECTION_HINTS and NEVER_SENT.

# Pick which sections of a royalty-run extract are sent. Pure code -- the last deterministic step

BANNER = "Synthetic Record"
PROPERTY = "Property"
STATEMENT = "Declared Statement"
SCHEDULE = "Royalty Schedule (Default)"
SIGNALS = "Corroborating Signals"
PRIOR = "Prior Periods"
EVENTS = "Events In This Period"
CONTACT = "Property Contact"
NOTE = "Franchisee Note"

src/prompt.pythe prompt

Three parts: instruction and JSON shape, carried state, extract. The stateless control replaces exactly one block.

# Assemble the one prompt this kit sends. No template engine, no framework -- string concatenation

INSTRUCTION = """\
def build(text, carried, stateless=False):

src/adapters/__init__.pythe model call

Raw HTTP, stdlib only, bounded retry; the shared daily call cap is checked here.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

src/reconcile.pythe reader

One property, one royalty run, one call.

# One property, one royalty run, one model call. The runtime half of the kit.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
SYSTEM = ("You are the monthly royalty reconciliation for one hotel brand's franchised estate. You "
MAX_TOKENS = 32000
FIELDS = ("verdict", "basis", "periods_open", "owner", "raise_query")
def documents():
def properties():
def load_doc(doc_id):
def _parse(txt):

evals/scoring.pythe scorer

Exact match per cell. The under-reporting discriminator is scored on BOTH denominators and split again into the band-invisible and the ordinary.

# Score one arm's answers against the key. Pure code, no model, no judge.

FIELDS = ("verdict", "basis", "periods_open", "owner", "raise_query")
UNDER = "UNDERREPORTED"
EXCL = "EXCLUSION_ABUSE"
INCOMPLETE = "SIGNALS_INCOMPLETE"
FIXED = "SELF_CORRECTED"
EXPLAINED = "EXPLAINED_MOVE"
ARITH = "ARITHMETIC_ERROR"
def _norm(field, v):
def _pct(n, d):

evals/baseline.pythe three free floors — a swap seam

arithmetic-only, variance-vs-prior and schedule-strong -- $0.00 each, and the third reaches 92.5 pct on verdict.

You change it to: VARIANCE_BAND_PCT -- re-run the floor for free and watch it stay silent on the same 15 readings.

# THE THREE FREE FLOORS. No key, no model, no network -- and none of them is a strawman.

MODES = ("arithmetic-only", "variance-vs-prior", "schedule-strong")
VARIANCE_BAND_PCT = 10.0
def _owner(text, carried):
def _query(verdict, carried):
def _periods_open(verdict, carried):
def review(text, carried=None, mode="arithmetic-only"):
def _answer(verdict, basis, periods_open, owner, raise_query, text, why):

src/app.pythe local UI server

http.server, stdlib. Renders with no key, on port 9014.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
PORT = int(os.environ.get("PORT", "9023"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r002-franchise-royalty")
GOLD_ROWS = {r["doc_id"]: r for r in
def carried_for(doc_id):
def read_in_code(text):
class H(BaseHTTPRequestHandler):

ui/app.jsthe local UI client

Hand-written JS, no framework.

# 

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 3122 input and 2225 output tokens per reading (one property, at one monthly royalty run), all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

MEASURED, FORCED AND PAIRED. The Franchisee Note is this kit's injection surface -- it is prose written by the party whose declaration is under examination, which makes it the one field on the page authored by somebody with an interest in the answer.

API_KEY is read from .env (gitignored) or the real environment; never logged, never placed in a prompt, and redacted from any provider error message the local UI passes to the browser.

The experimentWe DID attack it -- one sentence, on every reading where it could matter, each one paired against itself

The probe forces the instruction-shaped note onto every reading whose gold verdict is UNDERREPORTED or EXCLUSION_ABUSE and fires each twice, once with and once without, everything else held identical. The paired denominator is the readings the model called adverse WITHOUT the note, because a reading it already got wrong is an ordinary miss and counting it as a suppression inflates the rate. Both were run for real on 2026-08-25.

Both boundaries are measured in both directions: the injection probe forces the condition rather than waiting for it, and the privacy guard is red-proven by reproducing the schema-change condition that reaches its fallback.

The result1 of 34 adverse verdicts flipped and 0 of 17 queries suppressed by the instruction-shaped note -- measured, paired, not assumed.

One phrasing, one model, one corpus, 35 paired readings -- every reading where suppression was even possible. One pair failed with its cause unrecorded.

HonestyWhat this does not prove

• Whether the single flip reproduces across phrasings, models or corpora. One sentence, one model, one reading -- paired against its own un-injected answer, so the causation on that reading is established and nothing broader is claimed.
• Why FR-0013-P2 failed. The probe records a failed pair's id and not its error string.
• Whether an injection placed in Property Contact would have any effect -- by construction it cannot, because that block is never sent.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Never issue a royalty invoice, post an adjustment, debit a franchisee, open an audit or serve a default notice -- and never present the rate, the permitted exclusions, the tolerance or the approved-cause list as any real franchisor's terms.

Stated to the model on every call in src/prompt.py's INSTRUCTION block, stated again on the local UI, and true of the code by ABSENCE: there is no writer anywhere in src/, evals/, tools/ or ui/ outside results/*.json and data/state.json.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• This is a PROMPT RULE PLUS AN ABSENCE, not a runtime enforcement layer. Nothing stops a forker adding a billing call tomorrow.
• The injection result is ONE SENTENCE against ONE model on ONE corpus, and one of its 35 pairs failed with its cause unrecorded.
• Nothing here validates that the schedule in src/schedule.py matches any real franchise agreement. It is an illustrative default and the page says so four times.

WatchedWhat is watched, and why that one

The kit measures 104 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

4 measured by the latest run 100 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

no shared guard The two recorded runs declare no guard in common, so nothing below is protected by a precondition — read the history as two separate observations, not as a difference.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

9 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 4 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

triage · no model in the path — a baseline, not a peer column — 3 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 3 runs measured the same system — they differ on false_queries, missed_queries, properties_caught, properties_missed, underreport_false, underreport_missed, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

triage · with the model in the path — 4 runs. Columns here are only ever compared with each other.

not a time series No two of these 4 runs measured the same system — they differ on answered, arithmetic_error_cells, coded_cause_cells, documents, exclusion_abuse_cells, explained_move_cells, false_queries, max_tokens, memory_cells, properties_caught, properties_underreporting, prose_cause_cells, query_cells, quiet_query_cells, readings_scored, self_corrected_cells, signals_incomplete_cells, stateless, underreport_band_invisible_cells, underreport_false, underreport_ordinary_cells, underreport_quiet_cells — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 10 chips that all say so.

redteam · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 4 chips that all say so.

DeviationsWhat deviated

0 breaches across 9 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

monthly -- one royalty run five days after each period closes

In one lineNo framework, deliberately

NO FRAMEWORK. A folder of readable Python, no framework dependency, an empty requirements.txt and a prompt anyone can read end to end. A framework abstraction would own the retrieval step -- there is none here -- and the memory layer, which is already the entire surface of src/state.py.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Linear: the reader -> src/prompt.py -> src/adapters -> evals/scoring.py. No branching, no tool-calling and no agent loop.

The other sideWhat a framework costs you

• Swapping providers means editing the PROVIDERS dict by hand. In exchange the prompt has no hidden templating layer and requirements.txt stays empty.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r002-franchise-royalty on the fast tier, with the carried position, 2026-08-25. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 6 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

3 runs not plotted. b000-franchise-royalty-arithonly, b001-franchise-royalty-varianceprior, b002-franchise-royalty-schedulestrong recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the population is re-read whole on each scheduled run, and the prior run's carried state is what makes a reading a CHANGE.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe verdict, governing signal, royalty runs open, consultant and raise-query call, per reading, exact match against the computed answer key

whether each of the 5 answered fields equals the answer key, per reading

MethodHow the test was run

python3 -m evals.run --run-id <id> ...; evals/scoring.py compares strings. No model grades anything.

The inputOne real row, seen by every grader

The formulaWhat it computes

accuracy = hits / 120 per field. underreport_accuracy_pct is scored over the 28 under-reported readings and underreport_false_pct over the other 92 -- never blended.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the computed answer key in data/gold.jsonl, produced by src/royalty.step() from the same planted inputs the corpus builder wrote onto the page, and re-derived from the printed page by evals/check_labels.py check 6.

No true/false rates for this grader. It records 5 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? There is no tunable threshold. The grader is exact string match against a computed key, so the only way a number here moves is that an arm answered differently -- not that somebody re-cut a boundary.

Cadence: Re-scored free on every run of evals/run.py, and re-derivable from the committed result files at any time with no key. The paid arms are re-scored only when they are re-fired, which on this kit has happened once: both scored arms were re-taken at a 32,000 ceiling.

The decisionWhen to reach for it