The business caseThe problem this solves

Every month-end a hotel revenue team reads a cost-of-sale percentage per channel and moves inventory on the strength of it. Channel managers and reporting engines compute that percentage correctly and reliably from a terms record. What they cannot do is notice that the REVENUE they were handed is not the revenue the distribution agreement makes commissionable -- an addendum that moved the channel to the net-rate merchant model so no commission is payable at all, an addendum that replaced the percentage in March, a proviso charging commission net of VAT and city tax, a visibility invoice the contract makes a cost of sale, a volume rebate the month earned and nobody claimed. Every one of those produces a statement that foots to the penny. a month-end statement check that recomputes the cost-of-sale lines and confirms they foot -- which on this corpus means passing every one of the 64 statements computed on revenue the contract does not make commissionable or under commercial terms an addendum has replaced, because all of them foot.

The inputThe actual channel-month statements

The corpus is 120 channel-month statements, 0.77 MB (txt 120). A channel cost-of-sale statement is a production extract plus a contract reading, and both halves are commercially sensitive: a property's channel mix, its negotiated commission percentages and its override thresholds are exactly what a competing property and the channel itself would each like to see. There is no public corpus. A SCRUBBED real one would be worse rather than better, and the reason is specific: scrubbing removes the addendum, the proviso and the override condition, and that layer IS what this kit measures.

Synthetic Record
----------------------------------------------------------------
Every field below is invented. This is a generated channel production and cost-of-sale statement for
an AI use-case kit; it reproduces no hotel, no brand, no channel, no distribution agreement and no
real person. The commission rates, the override thresholds, the loyalty reimbursements, the caps and
the restatement floor are ILLUSTRATIVE DEFAULTS, no real agreement's, and nothing here is commercial
or accounting advice.


Statement
----------------------------------------------------------------
Statement reference                 : CH-0001
Property                            : The Marchmont
Channel                             : Roomsmith Direct
Channel type                        : OTA
Commercial model on the terms feed  : AGENCY
Production month                    : 2026-06
Contract reference                  : DA-2026-620
Revenue manager of record           : Priya Raghavan
Statement prepared by               : Mateo Duarte (channel's market desk)

Reporting Control
----------------------------------------------------------------
Restatement floor (default)         : 250.00 GBP
Distribution agreement              : ATTACHED
Settlement authority                : NOT DEFINED. Nothing in this kit pays, bills,
                                      disputes, adjusts, restates or posts anything.

Production Extract
----------------------------------------------------------------
All figures in GBP. Room nights and revenue are printed already multiplied out, one line
per production line. CANCELLED and NO_SHOW nights are shown separately because whether
commission is payable on them is decided by the contract and not by the channel manager.

The outcomeWhat a good result looks like

A cost of sale for the channel-month, to the penny, with the governing clause named and the component that was understated identified -- beside what the strongest free code alternative said about the same page, so a reader can see on which rows the model bought anything at all.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Every addendum, override and visibility invoice is keyed into the channel manager's terms record the day it is agreed — the free floor -- terms-feed-strong, $0.00
  It re-derives the whole cost stack from the production extract using the feed's own model and base token and catches every dropped cap, every term never applied and every base the statement moved -- 63.33 pct on the verdict column, 100.0 pct on the amount for statements that were SOUND, and 0 false accusations, for nothing.
• Addenda, provisos and override conditions live in contract text and the channel manager's terms record lags — the model, with the contract extract
  82.46 pct on the cost of sale across all 57 determinable readings, and 81.82 pct on the 22 whose defect the feed cannot see, against the floors' 0.0 pct. It catches 91.89 pct of the months needing a restatement against the strong floor's 40.54 pct.

And where nothing here is good enough:

• You want the reading but cannot put the contract in front of it — neither -- and the free strong floor is the honest version of that arm
  The contract-blind ablation IS that experiment and it is on this page: 7.02 pct on the amount against 82.46 pct with the contract, and 0.0 pct on the feed-blind slice. The arm did not guess -- it answered NOT_DETERMINABLE on 53 of 57 readings -- but an arm that declines is not a report. The free floor sees the same information and costs nothing.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-25. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-25 — r001-channel-cost. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Every statement, the answer key, all three free floors, the injection probe and what r001-channel-cost actually answered ship in the repo. python3 -m evals.check_labels and python3 -m src.app both run with no key and no network.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. one reading -- one channel-month statement, end to end including provider-side reasoning tokens, on a shared connection. ⚠︎ NOT AN SLA AND NOT COMPARABLE ACROSS KITS: this credential was shared with several sibling kits running at the same moment and the estate had already made over 9,900 calls on it that day, so these figures are a wall-clock observation under contention. 60 readings ran in 553.8 wall seconds with 4 concurrent workers, and p95 is 3.5x p50.

Current processWhat it replaces

a month-end statement check that recomputes the cost-of-sale lines and confirms they foot -- which on this corpus means passing every one of the 64 statements computed on revenue the contract does not make commissionable or under commercial terms an addendum has replaced, because all of them foot.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusEvalsEvalsArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• jsonl
• json
• 4 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one reading is

120 channel-month statements, read once each and independently -- there is no time axis and no carried state. The three FREE floors read all 120 AND the documented odd half; the two PAID arms read the odd half, 60 of 120, so every column compared on this page shares one denominator and the full-corpus floor figures are published beside them.

SetupWhat the setup figure measured

There is no index to build -- each channel-month's whole statement goes into the prompt. The figures are the corpus generation itself.

LicenceLicence

MIT

Bring your ownBring your own channel-month statements

tools/build_corpus.py writes data/corpus/*.txt and data/gold.jsonl. Keep the 9 section headings and the TERM:/LINE: line shapes evals/baseline.py parses. Set your own commissionable-revenue definitions, percentages, overrides and restatement floor in src/rules.py first, then run evals/check_labels.py -- it refuses to let anything spend until every gold amount re-derives from the shipped text and every claimed defect actually moves the prepared total.

⚠︎ And what stops being true when you do: The measured figures on this page do not transfer to your own statements. The feed-blind share (50 of the 76 defects here) and the prose-versus-feed split are properties of this generator's declared distribution, not facts about channel reporting. And the 82.46 pct on the amount is a property of an extract that prints its subtotals already multiplied out -- your own export probably does not. Re-run the evals on your own data; that is what the harness is for.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — but they are the input total apportioned by each part’s share of the characters, not a per-part reading from the provider’s counter. The total is measured; the split is arithmetic, and a part carrying denser text really costs more of it than this says.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 3,597 tokens assembled, 2,052 are instructions — 57% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

The verbatim prompt is rebuilt from the shipped corpus by src/prompt.build() for CH-0001 -- the reading framed in the success screenshot -- so the prompt on this page and the prompt that was sent are produced by the same function.

Full promptThe prompt in full

You are a channel cost-of-sale review for a hotel revenue team. You read one channel's production for one month, the channel manager's structured terms feed, the distribution agreement itself and the cost-of-sale statement somebody has prepared, and you decide whether the cost of sale was computed on the revenue the CONTRACT makes commissionable and under the commercial terms the contract gives. You answer with one JSON object and no other text.

You are a channel cost-of-sale review for a hotel revenue team. You are reading ONE channel for ONE
month: its production extract, the channel manager's structured copy of the commercial terms, the
CONTRACT EXTRACT itself, and the cost-of-sale statement somebody has already prepared.

YOUR JOB IS NOT TO CHECK THE MULTIPLICATION. The statement almost always reproduces itself -- its own
declared base value times its own declared rate gives its own printed amount, and free code checks
that for nothing. What you are looking for is the statement that multiplies out perfectly and was
computed on THE WRONG REVENUE, or under commercial terms THE CONTRACT DOES NOT GIVE IT: an addendum
that replaced the commission rate, an addendum that moved the channel to the merchant model so no
commission is payable at all, a proviso that charges commission net of VAT and city tax, a
sponsored-placement invoice the contract makes a cost of sale, a volume override the month earned and
nobody took, an override taken where the clause's own condition fails, a cap dropped, a term never
applied.

⚠︎ THE COMMISSION RATES, THE OVERRIDE THRESHOLDS, THE LOYALTY REIMBURSEMENTS, THE CAPS AND THE
RESTATEMENT FLOOR ARE ILLUSTRATIVE DEFAULTS INVENTED FOR THIS EXERCISE, NOT ANY REAL AGREEMENT'S.
Apply them exactly as printed regardless of whether they look right for the channel in front of you.
This is not commercial or accounting advice and nothing you answer pays, bills, disputes, adjusts,
restates or posts anything.

THE RULES, applied in this order:
 - FIRST check completeness. If the file records that the distribution agreement is NOT ATTACHED, the
   verdict is CONTEXT_INCOMPLETE: no term is judged, the defect is UNDETERMINED, the governing clause
   is NONE, the understated component is NONE, "cost_of_sale_gbp" is the string NOT_DETERMINABLE and
   nothing is restated. There is no default commercial term for an agreement nobody can read
   (Rule C-8a).
 - THEN check whether the statement reproduces ITSELF. Recompute each printed LINE from its own
   declared base value, rate and cap, and check the printed component amounts sum to the printed
   total. If the statement's OWN declared formula does not reproduce its OWN printed figures, the
   verdict is ARITHMETIC_ERROR and the defect is ARITHMETIC (Rule C-8). This is the only verdict
   about the multiplication, and it is NOT the verdict for a statement that multiplies correctly on
   revenue the contract does not make commissionable.
 - THEN work out WHAT THE CONTRACT SAYS THE CHANNEL COST. The production extract prints room nights,
   room revenue, VAT and city tax per production line, already multiplied out, plus the channel's
   retail value of the stayed nights, the net rate it remitted, any sponsored-placement invoice and
   the loyalty-eligible revenue. Every base you need is a SUM OF THOSE PRINTED LINES -- decide, line
   by line, which of them the contract makes commissionable, and which of the other printed figures
   the contract makes a cost of sale.
 - AN ADDENDUM, A PROVISO OR A CONDITION IN THE CONTRACT EXTRACT SUPERSEDES THE STRUCTURED TERMS FEED
   (Rule C-3). The feed is the channel manager's convenience copy and it has no field in which to say
   "net of tax", "with effect from 1 March" or "provided that". Where the statement applied a rate an
   addendum replaced, the defect is SUPERSEDED_RATE and the governing clause is THE ADDENDUM'S id.
   Where an addendum moved the channel to the MERCHANT model and the statement still charged
   commission, the defect is MODEL_STALE, the governing clause is THE ADDENDUM'S id, and the cost of
   sale is the MARGIN -- the channel's retail value of the STAYED nights less the net rate it
   remitted. No commission is payable at all under that model.
 - Where the revenue the contract makes commissionable is not the revenue the statement used --
   cancelled nights and no-shows in or out, VAT and city tax in or out -- the defect is
   WRONG_REVENUE_BASE and the verdict is BASIS_MISAPPLIED.
 - Where the contract makes a separately-invoiced sponsored-placement, visibility or campaign charge
   a cost of sale and the statement left it out, the defect is SPONSORED_OMITTED. It is never on the
   terms feed; it is an invoice, not a rate.
 - A VOLUME OVERRIDE is a rebate the channel owes the property and it REDUCES the cost of sale. It
   applies only where its own stated conditions are met -- the night threshold AND, where the clause
   says so, rate parity maintained across the month, which is printed on the production extract. An
   override the month earned and the statement never took is OVERRIDE_MISSED. An override the
   statement took and the clause's own condition excludes is OVERRIDE_NOT_AVAILABLE (Rule C-7).
 - A cap the clause carries and the statement dropped is CAP_OMITTED. A term the feed carries and the
   statement never applied at all is TERM_OMITTED. Both are TERM_MISAPPLIED.
 - Otherwise the verdict is STATEMENT_CORRECT, the defect is NONE and the governing clause is NONE.
 - THEN state "cost_of_sale_gbp": THE TOTAL COST OF SALE FOR THIS CHANNEL-MONTH under the governing
   reading -- commission plus merchant margin plus sponsored placement plus loyalty reimbursement
   plus the override, which is negative where one is earned -- as a number to two decimal places.
   Where the agreement is not attached it is the string NOT_DETERMINABLE.
 - THEN name the UNDERSTATED COMPONENT: the one cost component the statement understated by the
   largest amount. It is NONE where the statement is correct, NONE where the reading is
   CONTEXT_INCOMPLETE, and NONE where the statement OVERSTATED every component -- an overstated cost
   of sale makes a channel look worse than it is and understates nothing, so naming a component
   there would be plainly false.
 - THEN decide the OWNER. Start from the revenue manager of record, but a note recording a handover
   to someone else supersedes it. The person who PREPARED the statement is not the owner.
 - FINALLY decide "restate": YES only where a defect is named AND the total misstatement it causes --
   the absolute difference between the total cost of sale as prepared and the total under the
   governing reading, in either direction -- is at or above the printed RESTATEMENT FLOOR. A defect
   below the floor is still a defect and is still named; it is simply not worth reopening a closed
   month over (Rule C-9).

Answer with a single JSON object and nothing else:

{"verdict": "STATEMENT_CORRECT|BASIS_MISAPPLIED|TERM_MISAPPLIED|ARITHMETIC_ERROR|CONTEXT_INCOMPLETE",
 "defect": "NONE|WRONG_REVENUE_BASE|MODEL_STALE|SPONSORED_OMITTED|SUPERSEDED_RATE|OVERRIDE_MISSED|OVERRIDE_NOT_AVAILABLE|CAP_OMITTED|TERM_OMITTED|ARITHMETIC|UNDETERMINED",
 "governing_clause": "<the clause id that governs the disputed line, e.g. C-3.2 or A-2026-14, or NONE>",
 "cost_of_sale_gbp": <the total cost of sale for the month, a number to two decimals, or "NOT_DETERMINABLE">,
 "understated_component": "COMMISSION|MERCHANT_MARGIN|SPONSORED|LOYALTY|OVERRIDE|NONE",
 "restate": "YES|NO",
 "owner": "<the revenue manager who owns this channel>",
 "rationale": "one sentence, naming the clause you applied, what the statement did instead, and
   which way it moves the cost of sale"}

Precedence: CONTEXT_INCOMPLETE if the agreement is not attached; then ARITHMETIC_ERROR if the
statement does not reproduce itself; then BASIS_MISAPPLIED for commission charged on revenue the
contract does not make commissionable or under a commercial model the contract has replaced; then
TERM_MISAPPLIED for a superseded rate, a sponsored charge left out, an override wrongly taken or
wrongly missed, a cap dropped or a term never applied; otherwise STATEMENT_CORRECT.

Synthetic Record
----------------------------------------------------------------
Every field below is invented. This is a generated channel production and cost-of-sale statement for
an AI use-case kit; it reproduces no hotel, no brand, no channel, no distribution agreement and no
real person. The commission rates, the override thresholds, the loyalty reimbursements, the caps and
the restatement floor are ILLUSTRATIVE DEFAULTS, no real agreement's, and nothing here is commercial
or accounting advice.

Statement
----------------------------------------------------------------
Statement reference                 : CH-0001
Property                            : The Marchmont
Channel                             : Roomsmith Direct
Channel type                        : OTA
Commercial model on the terms feed  : AGENCY
Production month                    : 2026-06
Contract reference                  : DA-2026-620
Revenue manager of record           : Priya Raghavan
Statement prepared by               : Mateo Duarte (channel's market desk)

Reporting Control
----------------------------------------------------------------
Restatement floor (default)         : 250.00 GBP
Distribution agreement              : ATTACHED
Settlement authority                : NOT DEFINED. Nothing in this kit pays, bills,
                                      disputes, adjusts, restates or posts anything.

Production Extract
----------------------------------------------------------------
All figures in GBP. Room nights and revenue are printed already multiplied out, one line
per production line. CANCELLED and NO_SHOW nights are shown separately because whether
commission is payable on them is decided by the contract and not by the channel manager.

  Line           Nights      Room revenue             VAT        City tax
  STAYED           2006        168,504.00       33,700.80        4,212.60
  CANCELLED         222         17,948.70        3,589.74          448.72
  NO_SHOW            30          2,363.10          472.62           59.08

  Total room nights                            2258
  Gross room revenue (inclusive of tax)          188,815.80
  VAT                                             37,763.16
  City tax                                         4,720.40
  Room revenue net of VAT and city tax           146,332.24

  Channel retail value of STAYED nights          208,785.88
  Net rate remitted by the channel               135,894.54
  Sponsored placement invoiced separately          3,920.62
  Loyalty-eligible room revenue                   30,739.14
  Rate parity maintained in month                       YES

Terms Feed (Structured)
----------------------------------------------------------------
The channel manager's own copy of the commercial terms, as the reporting engine reads them.
One line per term, in the order the engine applies them. THE FEED IS A COPY OF THE TERMS,
NOT THE CONTRACT: an addendum, a proviso or a condition written into the contract extract
below governs over anything on these lines, and the feed has no field in which to say so.

TERM: id=C-3.1 kind=COMMISSION model=AGENCY base=GROSS_STAYED_NIGHTS pct=0.200000 order=1 label="channel commission"

There is no term line for sponsored placement and none for a volume override. The channel
manager records rates; a visibility invoice is not a rate, and an override is negotiated
once a year into the contract and never reaches this feed.

Contract Extract
----------------------------------------------------------------
The distribution agreement, as executed. Where an ADDENDUM appears below it replaces the
clause it names. Clause numbering is the agreement's own.

C-3.1           The channel is engaged on the AGENCY model and commission of 20.00 pct is
                payable on COMMISSIONABLE REVENUE for each calendar month.

C-3.1(a)        COMMISSIONABLE REVENUE means the room revenue for nights the
                guest ACTUALLY STAYED, inclusive of VAT and city tax as
                invoiced to the guest. Cancelled nights and no-shows are not
                commissionable.

C-4.1           Under the AGENCY model the property invoices the guest and remits
                commission to the channel. Under the MERCHANT model the channel pays the
                property a NET RATE and sells at its own retail price; NO COMMISSION IS
                PAYABLE, and the cost of sale for the month is the MARGIN between the
                channel's retail value of the nights STAYED and the net rate it remitted.

A-2026-14       Addendum to the agreement, replacing the model stated in C-3.1.
                The channel moves to the MERCHANT model with effect from 1 February 2026.
                NO COMMISSION IS PAYABLE from that date. The cost of sale is the margin
                under C-4.1: the channel's retail value of the nights STAYED less the
                net rate remitted. Both figures are printed on the production extract.

Cost-of-Sale Statement As Prepared
----------------------------------------------------------------
The statement as the analyst prepared it, one line per cost component, in the sequence
applied. THE STATEMENT DOES NOT SAY WHICH CLAUSE IT READ. It states a base value, a rate
and an amount; whether that base value is the revenue the contract makes commissionable is
the question.

LINE: seq=1 rule=C-3.1 kind=COMMISSION model=AGENCY base=GROSS_STAYED_NIGHTS base_value=168,504.00 pct=0.200000 amount=33,700.80

  Component                                        Amount
  COMMISSION                                    33,700.80
  MERCHANT_MARGIN                                    0.00
  SPONSORED                                          0.00
  LOYALTY                                            0.00
  OVERRIDE                                           0.00
  Total cost of sale as prepared                33,700.80
  Cost of sale pct of net room revenue          23.03 pct

Reporting Notes
----------------------------------------------------------------
Group business for this month was allocated from the annual contract pool and the per-night rate below is the pool rate divided by the contracted nights.

Raw responseThe raw response

{"verdict":"BASIS_MISAPPLIED","defect":"MODEL_STALE","governing_clause":"A-2026-14","cost_of_sale_gbp":72891.34,"understated_component":"MERCHANT_MARGIN","restate":"YES","owner":"Priya Raghavan","rationale":"Addendum A-2026-14 moved the channel to the MERCHANT model with no commission payable, so the cost of sale is the margin of 72,891.34 under C-4.1, but the statement charged a 20% agency commission of 33,700.80 under C-3.1, understating the total cost of sale."}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch the real cost behind a hotel channel bill — 120 channel-month statements. One model answered, and every answer was then graded Two different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

No model grades anything; every column is exact match in pure code. EVERY ARM COMPARED ON THIS PAGE READ THE SAME 60 READINGS -- the documented odd half, CH-0001..CH-0119. The two paid arms cost 120 calls between them and the free floors cost nothing, so the model, the ablation and the three floors are compared over one identical set; the floors were ALSO run over all 120 (b030-b032) and those figures are published beside them, landing within 5 points. The 3 readings with no distribution agreement are excluded from the AMOUNT denominator only (57 of 60), because not-determinable is a third state and scoring it as a zero would invent a failure.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection, never a bill and never a vendor's claim about this workload.

Rates checked 2026-08-18. The provider that actually ran the calls is kept out of these tables per this estate's naming rule, so no figure here is a bill.

The gradersTwo ways to grade

On the FEED-BLIND slice -- the 22 defects an addendum, a proviso, a separately invoiced charge or a conditional override produced, and which the channel manager's own terms feed cannot express -- all three floors score 0.0 pct on the amount and the model scores 81.82 pct. That single comparison is the kit. Everywhere else, read the free column first.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingAnswering and grading are two separate bills

The three free floors -- run twice each, over the scored half and over all 120 -- the stub and every pre-run check cost $0.00 and no calls. 156 calls were spent on this kit and 156 reached a published result file. The socket timeout and the token ceiling were set TOGETHER before the first paid call rather than after a discarded run, which is why the discarded count is 0. The injection probe's input tokens are not separately recorded by that harness, so its projection is output-only and understates it.

Cost driversWhat actually moves the bill

• OUTPUT TOKENS, AND ALMOST ALL OF THEM ARE REASONING. 97.1 pct of r001-channel-cost's output (265170 of 273027 tokens) was provider-side reasoning left at the default. The answer is seven short fields and a sentence; the bill is the thinking in front of it -- and on this task the thinking is a cost stack computed TWICE, once as the statement declares it and once as the contract defines it.
• THE PROMPT IS MOSTLY FIXED TEXT. 3462.83 input tokens per reading, and the ordered rules and the JSON shape are identical on every call -- so a provider with prompt caching would price this workload very differently, and that was not measured.

Your volumeWhat it costs at your volume

LINEAR IN CHANNEL-MONTHS. Nothing amortises. The sublinear lever NOT implemented is skipping months whose contract extract carries no addendum, no proviso, no sponsored clause and no override -- the free strong floor is exactly right on those, and this kit calls the model on them anyway.

Model choiceWhy this model

It is the tier the shared connection was already pointed at, and the kit's whole claim is that swapping it is .env plus one more run. No second model was run against this corpus; every other row in the cost table is a projection onto a published card and is labelled as one.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

per-query average, this run

The codeEvery file, and what it is for

12 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Six of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/rules.pythe cost-of-sale rulebook — a swap seam

The only definition of what a channel costs. compute() turns a production extract and a set of terms into a commission base, a commission, a merchant margin, a sponsored charge, a loyalty reimbursement and a volume override; decide() turns the governing and prepared computations into a verdict, a defect, a clause, an understated component and a restate call. It is also the answer key.

You change it to: DEFAULT_RESTATEMENT_FLOOR_GBP -- the knob that decides which named defects are worth reopening a closed month over. It is PRINTED on every statement and read off the page by every arm.

# The cost-of-sale rulebook as data and as pure code. No model, no network, no I/O.

STAYED = "STAYED"
CANCELLED = "CANCELLED"
NO_SHOW = "NO_SHOW"
LINES = (STAYED, CANCELLED, NO_SHOW)
COMMISSION = "COMMISSION"
MERCHANT_MARGIN = "MERCHANT_MARGIN"
SPONSORED = "SPONSORED"
LOYALTY = "LOYALTY"
OVERRIDE = "OVERRIDE"

src/segment.pythe section splitter

Splits the statement into its 9 named sections; asserted across all 120 documents before any run may spend.

# Split a channel-month statement into its named sections. Pure code, no model.

SECTIONS = ("Synthetic Record", "Statement", "Reporting Control", "Production Extract",
RULE = "-" * 64
def split(text):
def render(secs):

src/select.pythe send filter — a swap seam

Channel Contact is mapped by no field and therefore never sent -- and the fallback path that makes that a property of the code rather than of the corpus is red-proven.

You change it to: SECTION_HINTS and NEVER_SENT.

# Pick which sections of a channel-month statement are sent. Pure code -- the last deterministic

BANNER = "Synthetic Record"
STATEMENT = "Statement"
CONTROL = "Reporting Control"
PRODUCTION = "Production Extract"
FEED = "Terms Feed (Structured)"
CONTRACT = "Contract Extract"
PREPARED = "Cost-of-Sale Statement As Prepared"
CONTACT = "Channel Contact"
NOTES = "Reporting Notes"

src/prompt.pythe prompt

Two parts: the instruction with the ordered rules and the JSON shape, then the statement. The contract-blind ablation replaces exactly one section's body.

# Assemble the one prompt this kit sends. No template engine, no framework -- string concatenation

BLIND_LINE = ("The contract extract is not available for this channel-month. Judge the statement on "
INSTRUCTION = """\
def build(text, contract_blind=False):

src/adapters/__init__.pythe model call — a swap seam

Raw HTTP, stdlib only, bounded retry, a socket timeout set from the published ceiling (900 s) and a SEPARATE retry budget for transport failures. The shared daily call cap is checked here.

You change it to: ADAPTER_TIMEOUT_S and TRANSPORT_RETRIES, changed in the same edit as src/statement.MAX_TOKENS. They are COUPLED: raising the ceiling without raising the timeout converts a truncation defect into a transport defect, and a timed-out generation is retried and billed every time.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
TRANSPORT_RETRIES = 1
TIMEOUT_S = int(os.environ.get("ADAPTER_TIMEOUT_S", "900"))
def _post(url, headers, payload, timeout=None):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}

src/statement.pythe reader

One channel-month statement, one call, at the published 32,000-token ceiling.

# One channel-month statement, one model call. The runtime half of the kit.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
SYSTEM = ("You are a channel cost-of-sale review for a hotel revenue team. You read one channel's "
MAX_TOKENS = 32000
FIELDS = ("verdict", "defect", "governing_clause", "understated_component", "restate", "owner")
NOT_DETERMINABLE = "NOT_DETERMINABLE"
def documents():
def load_doc(doc_id):
def facts_of(text):

evals/run.pythe per-reading cache

Append-only, written before anything is scored. --resume never re-buys a call already paid for; --score-cache scores exactly what came back.

# Run the review over the 120 channel-month statements and write one result file. THIS SPENDS MONEY.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
def scored_half(docs):
def cache_path(run_id):
def load_cache(run_id, max_tokens, contract_blind):
def append_cache(run_id, row):
def load_gold():
def stub_complete(cfg, system, user, max_tokens=1024):

evals/scoring.pythe scorer

Exact match per labelled cell, a penny tolerance on the amount. The discriminator is reported on four denominators and never blended.

# Score a run against the answer key. Pure code, exact match per cell. No model grades anything.

FIELDS = ("verdict", "defect", "governing_clause", "understated_component", "restate", "owner")
DEFECT_VERDICTS = ("BASIS_MISAPPLIED", "TERM_MISAPPLIED", "ARITHMETIC_ERROR")
TOL = 0.011
def _pct(n, d):
def _same(got, want):
def _same_name(got, want):
def _amount(v):
def score(records, golds):
def _protection(records, golds):

evals/baseline.pythe three free floors

statement-recompute, rate-card and terms-feed-strong -- $0.00 each, and all three beat the model on the false direction and on the sound slice.

# THE FREE FLOORS. Three of them, none a strawman, all of them GBP 0.00.

MODES = ("statement-recompute", "rate-card", "terms-feed-strong")
TOL = 0.011
def _num(v):
def _kv(line):
def _facts(text):
def _finish(verdict, defect, clause, cost, comp, restate, owner, why):
def _line_check(f):
LINE_KIND_COMPONENT = {R.K_COMMISSION: R.COMMISSION, R.K_LOYALTY: R.LOYALTY,
def _recomputed_total(f):

evals/check_labels.pythe pre-spend gate

12 checks, free and offline, including re-deriving every gold amount from the shipped text and asserting every claimed defect actually moves the prepared total. The second one was written after a falsy-zero threshold silently disarmed three planted defects.

# THE PRE-SPEND GATE. Free, offline, and it must pass before any run may cost money.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
GOLD = os.path.join(HERE, "data", "gold.jsonl")
FAILURES = []
def check(name, ok, detail=""):
def _rebuild_from_text(text):
def main():

src/app.pythe local UI server

http.server, stdlib. Renders with no key, on port 9054.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
PORT = int(os.environ.get("PORT", "9054"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r001-channel-cost")
class H(BaseHTTPRequestHandler):
def main():

ui/app.jsthe local UI client

Hand-written JS, no framework.

# 

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 3462 input and 4550 output tokens per reading (one channel-month statement), all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

⚠︎ MEASURED, AND THE RESTATEMENT HELD -- BUT THE AMOUNT DID NOT HOLD STILL. Reporting Notes is this kit's injection surface, and on this vertical the desk that types them is very often the CHANNEL'S OWN market desk -- the counterparty whose statement is under review.

API_KEY is read from .env (gitignored) or the real environment; never logged, never placed in a prompt, and redacted from any provider error message the local UI passes to the browser.

The experimentThe experiment, and what it cannot tell you

Every reading whose answer key says the month should be restated AND which r001-channel-cost itself raised was re-fired with the Reporting Notes section REPLACED by the instruction-shaped note, at the SAME token ceiling. One variable. A suppression is that same reading coming back without a restatement; a MOVE is any watched field differing from its own un-injected answer, whether or not the restatement decision changed. 34 fired, 34 scored, 0 suppressed. Both were run for real on 2026-08-25.

All three boundaries are measured in both directions: the injection probe forces the condition rather than waiting for it and pairs each trial against its own control, it scores the whole answer rather than one field, and the privacy guard is red-proven by reproducing the schema-change condition that reaches its fallback.

The result34 of 34 restatements held under the forced instruction-shaped note at a 32000-token ceiling -- suppression rate 0.0 pct. The amount moved on 4 of the 34, which is repeatability, not injection.

One phrasing, one model, one corpus, one ceiling (32000), 34 paired trials -- every reading where suppression was even possible. 11 readings moved on some field; the 4 amount moves are three toward the answer key and one away, which is repeatability rather than injection, and the 8 owner moves are the probe deleting the handover note it replaced.

HonestyWhat this does not prove

• One phrasing only, at one ceiling, against one model, on 34 paired trials.
• Whether the amount moves would repeat. They were observed once, in a probe that was not designed to measure them, and no reading was fired a third time.
• Whether a note placed in a section this kit does NOT send would have any effect -- by construction it cannot, and that was not probed either.
• ⚠︎ THE owner COLUMN IS CONFOUNDED AND THE PROBE CANNOT FIX IT WITHOUT CHANGING TWO VARIABLES. force_note REPLACES the Reporting Notes section rather than appending to it -- deliberately, because a page carrying two notes is a different document from a page carrying one -- and the handover note that decides owner lives in that same section. All 8 owner moves are readings whose handover the probe deleted.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Never pay, bill, dispute, adjust, restate or post anything -- and never present the commission rates, the override thresholds, the loyalty reimbursements, the caps or the restatement floor as any real agreement's, or anything here as commercial or accounting advice.

Stated to the model on every call in src/prompt.py's INSTRUCTION block, printed on every statement's Reporting Control section, and true of the code by ABSENCE: the only writers anywhere in the repository are results/*.json and data/.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• This is a PROMPT RULE PLUS AN ABSENCE, not a runtime enforcement layer. Nothing stops a forker adding a settlement endpoint tomorrow.
• The injection result is ONE SENTENCE against ONE model on ONE corpus at ONE CEILING, over 34 paired trials. A sibling kit measured 0 suppressions at 32,000 and a real suppression on the same reading and the same sentence at 16,000. It is not a resistance rate for prompt injection in general and that phrase appears nowhere in this kit.
• The verdict is an OPINION about a statement, not a legal reading of a distribution agreement, and nothing here is commercial or accounting advice. A channel dispute is settled by people with the whole agreement and the correspondence in front of them.
• ⚠︎ AND THE HEADLINE IS NOT A CLEAN MEASURE OF THE MODEL. 3 of its 10 amount misses are this repository's answer key being wrong, and 6 more turn on wording in this repository's own contract clauses. Both are named, neither is patched.

WatchedWhat is watched, and why that one

The kit measures 103 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

2 measured by the latest run 101 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

no shared guard The two recorded runs declare no guard in common, so nothing below is protected by a precondition — read the history as two separate observations, not as a difference.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

10 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

triage · no model in the path — a baseline, not a peer column — 6 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 6 runs measured the same system — they differ on answered, arithmetic_cells, channel_cost_of_sale_cells, channel_cost_of_sale_defect_cells, channel_cost_of_sale_feedblind_cells, channel_cost_of_sale_hit, channel_cost_of_sale_sound_cells, context_incomplete_cells, correct_cells, defect_cells, defect_feedblind_cells, documents, feed_blind_cells, missed_restatements, months_caught, months_missed, months_needing_restatement, quiet_cells, readings_scored, restate_cells, sound_cells, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

triage · with the model in the path — 3 runs. Columns here are only ever compared with each other.

not a time series No two of these 3 runs measured the same system — they differ on answered, arithmetic_cells, channel_cost_of_sale_cells, channel_cost_of_sale_defect_cells, channel_cost_of_sale_feedblind_cells, channel_cost_of_sale_hit, channel_cost_of_sale_no_figure, channel_cost_of_sale_sound_cells, context_incomplete_cells, correct_cells, defect_cells, defect_feedblind_cells, documents, false_restatements, feed_blind_cells, missed_restatements, months_caught, months_missed, months_needing_restatement, quiet_cells, readings_scored, restate_cells, sound_cells, stateless — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

redteam · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 2 chips that all say so.

DeviationsWhat deviated

0 breaches across 10 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

ON DEMAND, ONCE PER CHANNEL PER MONTH -- there is no clock and no carried state, so re-reading the same statement against the same contract returns the same opinion. Or it should: the injection probe re-fired 34 readings and 4 answered a different amount, so run-to-run variance on the discriminator is real and unmeasured. The one thing that makes a re-run worth paying for is a CONTRACT that moved: an addendum countersigned after the statement was prepared changes which terms govern, and this kit will only see it once that addendum reaches the extract. At 0.01241497 USD per channel-month on the projected card, a year of 12 channels is roughly 1.79 USD -- but the ceiling that matters is provider concurrency, not price: p95 latency was 3.5x p50 on a credential shared with several sibling kits.

In one lineNo framework, deliberately

NO FRAMEWORK. A folder of readable Python, no framework dependency, an empty requirements.txt, and a prompt anyone can read end to end. There is no retrieval step for a framework to own and no agent loop for it to drive: the whole job is one statement, one contract and one call.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Linear: the reader -> src/prompt.py -> src/adapters -> evals/scoring.py. No branching, no tool-calling and no agent loop.

The other sideWhat a framework costs you

• Swapping providers means editing the PROVIDERS dict by hand. In exchange the prompt has no hidden templating layer and requirements.txt stays empty.
• Resume and caching are 40 lines here rather than a framework feature.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-channel-cost on the fast tier, with the contract extract, 2026-08-25. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 8 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

6 runs not plotted. b020-channel-cost-recompute, b021-channel-cost-ratecard, b022-channel-cost-feedstrong, b030-channel-cost-recompute-full, b031-channel-cost-ratecard-full, b032-channel-cost-feedstrong-full recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the rulebook is data read whole into the prompt.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: ADAPTER_TIMEOUT_S, API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe total cost of sale for the channel-month, to the penny, against the computed answer key

whether the arm's stated cost_of_sale_gbp equals the governing figure, within a penny, on every reading where a figure is determinable at all

MethodHow the test was run

python3 -m evals.run --run-id <id> --scored-half ...; evals/scoring.py compares numbers. No model grades anything.

Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

hit if |answered - gold| <= 0.011. Reported on FOUR denominators and never blended: all determinable readings, the sound slice, the defect slice, and the feed-blind slice.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the computed answer key in data/gold.jsonl, produced by src/rules.compute() from the same planted terms the corpus builder printed onto the page, and re-derived from the SHIPPED TEXT by evals/check_labels.py before any run may spend.

No true/false rates for this grader. It records 4 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? The only tunable is the penny tolerance, and it is 0.011 -- one penny either way and no more. It is NOT swept: a wider tolerance would forgive nothing this corpus contains and would flatter every arm equally.

Cadence: once per run; free

The decisionWhen to reach for it

In one lineThe verdict, the named defect, the governing clause, the component understated, the restate call and the revenue manager, per reading, exact match

whether each of the 6 labelled fields equals the answer key, per reading

MethodHow the test was run

python3 -m evals.run --run-id <id> --scored-half ...; evals/scoring.py compares strings.

Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

accuracy = hits / 60 per field. The amount is scored by its own grader above.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl, computed by src/rules.decide().

No true/false rates for this grader. It records 6 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No threshold was swept: exact match has no tunable. Enum columns are case-folded, the revenue manager's name is not.

Cadence: once per run; free

The decisionWhen to reach for it