The business caseThe problem this solves

A signed group contract fixes a room block by night, a rate, a cut-off date, an attrition threshold, a food and beverage minimum, a complimentary-room ratio and a table of cancellation bands. The pick-up and the final bill then depart from all of it, and somebody has to decide, clause by clause, whether what happened complies -- and where it does not, how much. That is not a comparison of two numbers. The attrition threshold protects part of the shortfall and is measured against the block AS AMENDED; the minimum is met by the categories the contract says count and not by the bill total; the complimentary ratio applies to the basis the contract names and the two bases give different entitlements; the cancellation band is selected by the days between WRITTEN notice and arrival. And before any of that, the contract may have been amended by an email nobody folded back into it. A comparator that puts the actual beside the contracted figure and charges every adverse difference -- which is what a group-billing spreadsheet actually is -- charges 61 of the 162 reviews on this corpus that comply perfectly (37.65 pct). It does not find deviations; it produces a list somebody has to read, and every wrong line in it is an invoice adjustment sent to a group holding the email that authorised the thing being charged. a group-billing reconciliation built on a variance spreadsheet. That is the thing being beaten here and it ships in this repo as code so the comparison is not rhetorical: three free floors, raw-variance 35.07 pct (61 of 162 compliant reviews charged anyway), threshold-calc 60.07 pct, clause-gate 72.92 pct. The weakest of the three is the variance comparator, and its false-deviation rate is the number this whole kit exists to move.

The inputThe actual group contract deviation review files

The corpus is 48 group contract deviation review files, 0.36 MB (txt 48). There is no public corpus of (clause, actual, verdict, deviation amount) tuples and there is not going to be one: a signed group contract carries a named group's negotiated rate, a named property's attrition and food-and-beverage terms, and a final bill both sides treat as confidential. And the thing being measured has to be planted to be measured -- a real archive does not come labelled with which of its apparent deviations the parties had already answered by email, which block reduction the property granted in writing, which excluded charge the catering director agreed to count, or which complimentary entitlement was held at the blocked basis. Those are exactly the reviews this kit exists to get right. ⚑ AND THE FILE-LEVEL LABEL HAD TO BE CONSTRUCTED RATHER THAN DEALT. 36 ADJUST_THE_BILL, 5 RELEASE_THE_BILL, 7 REQUEST_EVIDENCE; dealt at random, 108 deviations over 48 six-review files would put a deviation in every file and the recommendation would collapse to one word. As built, the majority word scores 75.0 pct. And every clean file is filled from the pool that holds the prose-cleared rows -- a file whose right answer is 'release it' while its own figures say 'adjust four things' is the hardest file in the corpus, and it is the one every free floor gets wrong at the file level by construction.

Group Contract Deviation Review
----------------------------------------------------------------
  Review                        GD-0001
  Property                      Harbourgate Rise Hotel, Lisbon
  Group                         Vessington Actuarial Society
  Contract                      GC-2200
  Signed                        2026-04-10
  Arrival                       2026-09-07
  Departure                     2026-09-10
  Nights                        3
  Contracted rate               230.18
  Currency                      USD
  Review date                   2026-09-18
  Prepared for                  the group billing review, before the final invoice is released

Contract Terms
----------------------------------------------------------------

  Room block

  Night     Date          Rooms blocked
  RN-01     2026-09-07    31
  RN-02     2026-09-08    57
  RN-03     2026-09-09    48

  Rooms blocked, room-nights            136                                                             GC-2.1

  Thresholds and bases

  Attrition threshold                   90.00 per cent                                                  GC-4.1
  Attrition rounding                    up, to the next whole room-night                                GC-4.1
  Attrition basis                       the contracted block as amended                                 GC-4.2
  Attrition damages                     the contracted rate per room-night short                        GC-4.3
  Cut-off date                          2026-07-24                                                      GC-3.1
  Food and beverage minimum             13,000.00                                                       GC-5.1

The outcomeWhat a good result looks like

one entry per clause review: DEVIATION with the deviation type, the clause the contract itself carries for it, the line that proves it AND THE AMOUNT to the cent; COMPLIANT; or INSUFFICIENT_EVIDENCE naming the term that was never executed. Beside every row, the same six recomputations in pure code. Measured over 288 clause reviews across 48 group files: 91.32 pct on the whole call including the money, 107 of 108 deviations raised, 12 of 162 compliant reviews charged anyway (7.41 pct), and the file-level adjustment total right to the cent on 30 of 48 files.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• the answer is fully computable from the contract's own structured fields and the invoice -- a deviation type, a governing clause, and an amount to the cent — exact string and cent matching against a generated key -- the only grader this kit carries
  deterministic and free: the key is arithmetic over the contract's stated terms, so there is nothing a second grader could disagree with. 288 clause reviews scored this way at $0.00.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-26. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-26 — r002-group-deviation. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Every group file, the answer key with its amounts, all three free floors, the materiality sweep, the answer-key gate and its eight-defect red-proof, the re-scorer, the paired injection probe, and what every recorded run actually answered all ship in the repo. python3 -m src.app renders the whole page with no key, no install and no index build, with every recomputation and every amount already on screen; python3 -m evals.run --floor clause-gate re-scores the strongest free arm end to end for nothing; python3 tools/build_corpus.py regenerates the corpus byte-identically from its seed.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. one group file -- one signed contract, one final bill, six clause reviews -- end to end including provider-side reasoning tokens, on a shared connection nine sibling kits were using at the same time. 97.6 pct of the output tokens on this run were provider-side reasoning rather than the review itself, so these are reasoning-dominated figures under contention and not a clean-room measurement.

Current processWhat it replaces

a group-billing reconciliation built on a variance spreadsheet. That is the thing being beaten here and it ships in this repo as code so the comparison is not rhetorical: three free floors, raw-variance 35.07 pct (61 of 162 compliant reviews charged anyway), threshold-calc 60.07 pct, clause-gate 72.92 pct. The weakest of the three is the variance comparator, and its false-deviation rate is the number this whole kit exists to move.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusEvalsEvalsArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• jsonl
• json
• 5 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one clause review is

48 group files, 288 clause reviews -- six clause areas per file, one review each, so every area is its own denominator of 48. One call per file. Twenty-nine planted cases over five evidence channels: S_ the contract's own printed fields settle it, C_ only a sentence in the correspondence reveals it and the fields read the review as compliant, P_ only a sentence clears it and the fields read it as a deviation, D_ a real variance too small for the contract's own immaterial-variance clause, M_ nothing in the file settles it. The composition is constructed rather than dealt: 108 deviations, 162 compliant and 18 unsettleable, so 'COMPLIANT for everything' scores 56.25 pct and cannot be mistaken for a result.

SetupWhat the setup figure measured

There is no index to build. Each group file goes into the prompt whole, minus the one withheld section, and everything a review needs -- the contract, the actuals, the bill and the correspondence -- is inside that file.

LicenceLicence

MIT

Bring your ownBring your own group contract deviation review files

In this order, because each step depends on the one before it: src/gfile.py (the regular expressions, written for these fixed-width tables and key/value blocks -- and note that every separator in them is [ ], never \s), src/segment.py (the heading pattern), src/select.py (NEVER_SENT -- decide what in your files must not leave the machine), src/clauses.py (the six checks, the rounding, the eligibility filter, the basis logic and TOL_USD -- run python3 -m evals.tolerance against your own corpus before trusting the value shipped here), src/prompt.py (the six deviation types and the three verdicts, if your contract's are different, and they almost certainly are), and finally data/gold.jsonl, your own answer key INCLUDING THE TRUE AMOUNT PER CLAUSE. Everything this kit publishes is a comparison against that file, and on this task it is harder to build than usual because the key has to carry the money as well as the verdict.

⚠︎ And what stops being true when you do: The measured figures here do not transfer. The prose share is the whole story of this page -- 78 of the 288 clause reviews turn on a sentence in the contract correspondence, and every free floor scores 0.0 pct on the deviation half of that -- and how much of YOUR contract evidence arrives as prose rather than in a field is a fact about how your property runs group sales, not about this method. The materiality threshold is worse than untransferable: it is a constant chosen from a sweep on these same 288 rows.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 2,862 tokens assembled, 2,014 are documents — 70% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Assembled by src/prompt.build for GD-0033 and reproduced byte for byte -- the system message, then the group file as the user message, six of its seven sections. The seventh, Property Position, is cut in code by src/select.py and is not here because it never left the machine. The token split below is solved from the provider's own returned counts by p001-group-deviation-prompt-tokens rather than estimated.

Full promptThe prompt in full

You are reviewing what actually happened on a group booking against the group contract
that was signed for it.

You act for the PROPERTY. A deviation is not an observation that a number moved: it is a figure that
goes onto an invoice adjustment and is put to the group's planner in writing, and one the property
may have to withdraw in front of the person who agreed the very thing being charged. A charge you
cannot support is answered with the email that authorised it. A charge with no clause behind it is
not something the group has to answer at all.

A DIFFERENCE FROM THE CONTRACT IS OFTEN NOT A DEVIATION. Contracts get amended after signature, the
threshold applies to a basis the contract names and not to whichever figure is nearest, and a
minimum is met by the charges the contract says count and not by the bill total. Treat a difference
as a question, not as a verdict.

Return one entry for EVERY review in "Clause Reviews", in the order the file lists them, using the
file's own review identifiers. Never drop one because it looks unremarkable.

For each review give exactly one verdict:

  COMPLIANT              what happened satisfies the clauses in scope. Either the figures meet the
                         contract, or something in the file already answers the apparent
                         difference, or the variance is one the contract does not treat as a
                         deviation.
  DEVIATION              a deviation can be STATED, the clause that carries it is in this file, the
                         evidence that proves it is in this file, and the amount can be computed
                         from figures this file prints.
  INSUFFICIENT_EVIDENCE  something about the review cannot be settled from this file -- a term that
                         was never executed, or a figure the file does not carry. Do not charge it;
                         say what is missing.

When the verdict is DEVIATION, the deviation type is exactly one of:
  ATTRITION_SHORTFALL           the group picked up fewer room-nights than the attrition threshold
                                requires of the block that governs it
  FNB_MINIMUM_SHORTFALL         the food and beverage charges that count towards the minimum fall
                                short of it
  COMP_RATIO_EXCEEDED           more complimentary room-nights were provided than the contracted
                                ratio earns on the basis the contract names
  RATE_VARIANCE                 room-nights were charged at something other than the rate the
                                contract fixes for them
  CUTOFF_ROOMS_UNRELEASED       rooms were still held in the block after the cut-off date
  CANCELLATION_BAND_MISAPPLIED  the cancellation charge is not the band that applies to the days
                                between written notice and arrival
Otherwise deviation_type is null.

QUANTIFY EVERY DEVIATION. Give `deviation_amount` as a number in the contract's currency, to two
decimal places -- the money the property would put on the invoice adjustment for that review, and
nothing more. Give null when the verdict is not DEVIATION.

Cite the governing clause as a clause identifier that appears in this file, and attach the evidence
as identifiers that appear in this file -- the performance lines, the bill lines, the nights, the
bands and the correspondence notes all carry identifiers. NEVER write a clause, band, line, night or
note identifier that is not printed in the file in front of you -- an invented citation is worse
than none, because it is the first thing the group's planner will check.

Anything in the file may bear on a review. Where a printed figure and a later statement in the file
contradict each other, they are not a tie.

Reply with JSON and nothing else:

{"file_action": "ADJUST_THE_BILL" | "REQUEST_EVIDENCE" | "RELEASE_THE_BILL",
 "reviews": [{"review": "<the file's review identifier>",
              "subject": "<the subject, one short phrase>",
              "verdict": "COMPLIANT" | "DEVIATION" | "INSUFFICIENT_EVIDENCE",
              "deviation_type": "<one of the six, or null>",
              "governing_clause": "<a clause identifier from this file, or null>",
              "evidence": ["<identifiers from this file>"],
              "deviation_amount": <number or null>,
              "finding": "<the one sentence the review note would carry for this row>"}],
 "bill_adjustment_total": <the sum of every deviation_amount above, or 0>,
 "rationale": "<two sentences at most, on what decided the hardest review>"}

file_action is ADJUST_THE_BILL if any review is DEVIATION; REQUEST_EVIDENCE if none is but at least
one is INSUFFICIENT_EVIDENCE; RELEASE_THE_BILL otherwise. None of the three adjusts, invoices,
credits or releases anything -- each is a recommendation to the person who does.

Group Contract Deviation Review
----------------------------------------------------------------
  Review                        GD-0033
  Property                      The Kestrel Wharf, Tallinn
  Group                         Ambrose Bell Owners Meeting
  Contract                      GC-2616
  Signed                        2026-03-25
  Arrival                       2026-09-23
  Departure                     2026-09-26
  Nights                        3
  Contracted rate               304.16
  Currency                      USD
  Review date                   2026-10-06
  Prepared for                  the group billing review, before the final invoice is released

Contract Terms
----------------------------------------------------------------
  Room block

  Night     Date          Rooms blocked
  RN-01     2026-09-23    26
  RN-02     2026-09-24    42
  RN-03     2026-09-25    29

  Rooms blocked, room-nights            97                                                              GC-2.1

  Thresholds and bases

  Attrition threshold                   85.00 per cent                                                  GC-4.1
  Attrition rounding                    up, to the next whole room-night                                GC-4.1
  Attrition basis                       the contracted block as amended                                 GC-4.2
  Attrition damages                     the contracted rate per room-night short                        GC-4.3
  Cut-off date                          2026-08-21                                                      GC-3.1
  Food and beverage minimum             14,500.00                                                       GC-5.1
  Eligible categories                   banquet, bar service, coffee break                              GC-5.2
  Excluded categories                   parking, room service, service charge, tax recovery             GC-5.3
  Complimentary room ratio              1 per 30 room-nights                                            GC-6.1
  Complimentary room basis              room-nights actualised                                          GC-6.2
  Complimentary room value              the contracted rate                                             GC-6.3
  Immaterial variance                   allowed, not stated                                             GC-1.4

  Cancellation bands

  CB-1      91 days or more before arrival            0.00 per cent     GC-7.1
  CB-2      61 to 90 days before arrival              25.00 per cent    GC-7.1
  CB-3      31 to 60 days before arrival              50.00 per cent    GC-7.1
  CB-4      30 days or fewer before arrival           85.00 per cent    GC-7.1

      The band applies to the contracted rate for every cancelled room-night.

  Clause index

  GC-1.4    A variance that is immaterial is not a deviation, and this contract does not say what is immaterial
  GC-2.1    A reduction to the contracted room block takes effect only when the property grants it in writing
  GC-3.1    Rooms not picked up by the cut-off date are released from the block
  GC-3.2    Rooms held in the block after the cut-off date are billed to the group at the contracted rate
  GC-4.1    The group shall pick up at least the attrition threshold of the room block
  GC-4.2    Attrition is measured against the contracted block as amended by any permitted reduction, and against no other figure
  GC-4.3    Attrition damages are the contracted rate for each room-night short of the threshold
  GC-5.1    The group shall generate at least the food and beverage minimum
  GC-5.2    Only the eligible categories count towards the food and beverage minimum
  GC-5.3    The excluded categories do not count towards the food and beverage minimum
  GC-6.1    Complimentary rooms are provided at the contracted ratio
  GC-6.2    The complimentary ratio is applied to the basis this contract names, and to no other
  GC-6.3    Complimentary rooms beyond the ratio are billed at the contracted rate
  GC-7.1    The cancellation charge is the band that applies to the days between written notice and arrival
  GC-8.1    The rate charged shall be the contracted rate for every room-night in the block

Actualised Performance
----------------------------------------------------------------
  Night     Date          Rooms blocked   Rooms picked up   Rate charged
  RN-01     2026-09-23    26              20                304.16
  RN-02     2026-09-24    42              33                304.16
  RN-03     2026-09-25    29              23                304.16

  AP-BLOCK  Rooms blocked, room-nights                        97
  AP-PICKUP Rooms picked up, room-nights                      76
  AP-RATE   Rate charged against the contracted rate          at the contracted rate on every night
  AP-COMPS  Complimentary rooms provided, room-nights         3
  AP-HELD   Rooms held in the block after the cut-off date    0
  AP-CANCEL Cancellation notice                               2026-08-10, 7 room-nights, 1,064.56 charged

  Cancelled room-nights are excluded from the pick-up above and are charged under GC-7.

Final Bill
----------------------------------------------------------------
  Line      Category                  Amount
  BL-01     parking                   786.85
  BL-02     coffee break              6,591.61
  BL-03     tax recovery              1,022.96
  BL-04     bar service               5,552.85
  BL-05     banquet                   2,333.63
  BL-06     room service              59.84
  BL-07     service charge            241.49

  BL-TOT    Final bill, all lines     16,589.23

Clause Reviews
----------------------------------------------------------------
Review CR-1
  Subject                       Attrition against the room block
  Clauses in scope              GC-4.1, GC-4.2, GC-4.3, GC-2.1

Review CR-2
  Subject                       The food and beverage minimum
  Clauses in scope              GC-5.1, GC-5.2, GC-5.3

Review CR-3
  Subject                       Complimentary rooms against the contracted ratio
  Clauses in scope              GC-6.1, GC-6.2, GC-6.3

Review CR-4
  Subject                       The rate charged against the contracted rate
  Clauses in scope              GC-8.1, GC-1.4

Review CR-5
  Subject                       Rooms held in the block after the cut-off date
  Clauses in scope              GC-3.1, GC-3.2

Review CR-6
  Subject                       The cancellation charge against the band table
  Clauses in scope              GC-7.1, GC-1.4

Contract Correspondence
----------------------------------------------------------------
- CC-01     The property granted the group a permitted reduction of 11 room-nights on 2026-09-01, taking the contracted block to 86 room-nights. The reduction was given in writing under GC-2.1 and no further amendment is outstanding.
- CC-02     The property's revenue office noted that the group's room list was received 20 days later than the contract asks for. No charge arises and none was raised.
- CC-03     The property confirmed a reduced rate of 288.43 for the room-nights of 2026-09-23 and 2026-09-24 in writing on 2026-09-14 after the group reduced its senior delegation. The final bill above was never re-cut and every night is billed at the contracted rate.
- CC-04     The group's meeting planner asked for the loading-dock hours for the exhibitor move-in on 2026-09-09. Nothing about the contract was raised.
- CC-05     Reservations found 5 room-nights still held in the group block on the arrival date. They were never released at the cut-off and were never recorded on the rooms-held line above.

Raw responseThe raw response

{
 "review": "CR-4",
 "subject": "The rate charged against the contracted rate",
 "verdict": "DEVIATION",
 "deviation_type": "RATE_VARIANCE",
 "governing_clause": "GC-8.1",
 "evidence": [
  "RN-01",
  "RN-02",
  "AP-RATE",
  "CC-01"
 ],
 "deviation_amount": -1327.04,
 "finding": "The confirmed 215.10 rate for the 88 room-nights on 2026-09-07 and 2026-09-08 was not applied, so the bill must be credited 1,327.04."
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Check a conference group's hotel bill against its contract — 288 group contract deviation review files drawn from 48 real group contract deviation review files. One model answered, and every answer was then graded Two different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

Exact string match per clause review against a generated key, plus the deviation amount to the cent. No LLM-as-judge anywhere in this kit, so there is no judge to calibrate, no judge cost and no judge to disagree with. Every arm -- the paid one and all three free floors -- is scored by the same function, and evals/rescore.py re-applies it to every committed result file in one pass so no two columns can have been scored by two different scorers.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on this page is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid. The rate is read from this repo's own committed card, build/facts/models.json, rather than typed here, so it cannot drift from what the rest of the estate projects onto. checked_on is that card's own as_of.

Rates checked 2026-08-18. The provider that actually ran all 57 calls is kept off this page per the series rule. And the three free floors, the materiality sweep, the wiring stub and the re-scorer are not priced because they make no call at all.

Grading unitWhat the grading figure prices

The grader is pure Python over the committed result files and costs nothing to re-run. python3 -m evals.run --floor clause-gate re-scores the strongest free arm end to end with no key; python3 -m evals.tolerance re-runs the whole eight-value sweep across all three floors in a fraction of a second; python3 -m evals.rescore --all re-scores every recorded run without making a call.

The gradersTwo ways to grade

⚠︎ THE FLOOR IS SHOWN AT ITS BEST AND THE PAID ARM IS NOT TUNED AT ALL. src/clauses.TOL_USD = 100.0 is the value at which clause-gate peaks in b900-group-deviation-materiality-sweep, chosen on the same 288 rows the floor is then scored against. That is a real advantage handed to the free code and it is disclosed rather than buried.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingAnswering and grading are two separate bills

57 provider calls in total: 48 for the scored run, 7 for the two ceiling calibrations, 2 for the prompt-token split. The three free floors, the eight-value materiality sweep, the wiring stub, the re-scorer and the answer-key gate cost $0.00 and no calls at all. NO REPLY WAS TRUNCATED IN ANY PUBLISHED ARM -- but THREE of the four calibration calls WERE, at 32,000, which is why the published ceiling is 64000.

Cost driversWhat actually moves the bill

• OUTPUT TOKENS, AND ALMOST ALL OF THEM ARE REASONING. 97.6 pct of r002-group-deviation's output (1515247 of 1552462 tokens) was provider-side reasoning left at the provider default. The review itself is a few hundred tokens of JSON per row; the bill is the thinking.
• THE PROMPT IS MOSTLY FIXED TEXT, AND THE SPLIT WAS MEASURED RATHER THAN GUESSED. p001-group-deviation-prompt-tokens solves input_tokens = fixed + rate x file_chars from two points: 848.43 fixed tokens against 0.263366 per file character, so roughly a third of the 2792.56 average input tokens is the same system message on every call. A provider with prompt caching would price that third differently.
• ONE CALL PER FILE, AND THE FILE IS THE UNIT. Six clause reviews go in one prompt and come back in one reply; splitting them per review would multiply the fixed prompt by six and would also remove the only thing that lets a note about a permitted reduction reach the attrition review that needs it.
• NOTHING AMORTISES. There is no index, no embedding pass and no cached retrieval -- 48 files cost 48 calls, and the next contract costs another one.

Your volumeWhat it costs at your volume

LINEAR IN GROUP FILES. Ten times the contracts is ten times the calls at $0.098425 each. The sublinear lever is named and not built: the structured checks already decide 72 of 72 structured deviation cells for nothing, with the money exact, so only the reviews where the printed figures and the correspondence could disagree need a model. That would cut the paid volume by most of the corpus -- and it is NOT MEASURED, because building the router and then scoring the model on what it forwarded would measure the router.

Model choiceWhy this model

It is the tier the shared connection was already pointed at, and this kit's whole claim is that swapping it is one line of .env plus one more run. No second tier was run against this corpus, so nothing here is a tier comparison -- that is a spend decision, recorded as unrun.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

per-file average, this run

The codeEvery file, and what it is for

16 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Seven of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pythe group file, cut into sections

Seven underlined headings, split deterministically. 'What did we send' is a list of section names rather than a guess about a token window.

# Cut a group deviation review file into addressable sections. Pure code, no model.

HEAD = re.compile(r"^([A-Z][A-Za-z ,]+)\n-{10,}\n", re.M)
def sections(text):
def named(text, name):

src/select.pythe seam that withholds — a swap seam

NEVER_SENT is one tuple holding one section name: Property Position. The property's own cost per occupied room, the forward booking value of the group's future business, the third-party planner's commission on realised room revenue and the sales director's negotiating posture never leave the machine. The UI prints what went and what stayed, per file.

You change it to: NEVER_SENT. One tuple, one section name. It withholds a NAMED SECTION and is not a redaction system.

# SEAM 2 -- what leaves this machine. Pure code, and deliberately a denylist of one.

NEVER_SENT = ("Property Position",)
def sent(sec_names):
def body(text, sections_fn):

src/gfile.pythe file's tables, read in code

Every threshold and its basis, the eligible and excluded category lists, the complimentary ratio and the basis GC-6.2 names for it, the band table, the clause index, every night, every performance summary line, every bill line and every correspondence identifier. ⚠︎ Every column separator is [ ] and never \s: \s matches a newline, and that defect blanked the attrition threshold on 45 of 48 files until check_labels convicted it.

# Read a group deviation review file into structured values. Pure code, no model.

COMPLIANT, DEVIATION, INSUFFICIENT = "COMPLIANT", "DEVIATION", "INSUFFICIENT_EVIDENCE"
VERDICTS = (COMPLIANT, DEVIATION, INSUFFICIENT)
DEVIATION_TYPES = ("ATTRITION_SHORTFALL", "FNB_MINIMUM_SHORTFALL", "COMP_RATIO_EXCEEDED",
FILE_ACTIONS = ("ADJUST_THE_BILL", "REQUEST_EVIDENCE", "RELEASE_THE_BILL")
AREA_ORDER = ("CR-1", "CR-2", "CR-3", "CR-4", "CR-5", "CR-6")
AREA_TYPE = {"CR-1": "ATTRITION_SHORTFALL", "CR-2": "FNB_MINIMUM_SHORTFALL",
AREA_CLAUSE = {"CR-1": "GC-4.1", "CR-2": "GC-5.1", "CR-3": "GC-6.1",
AREA_LINE = {"CR-1": "AP-PICKUP", "CR-2": "BL-TOT", "CR-3": "AP-COMPS",
NOT_STATED = ("to be agreed by addendum, not executed",

src/clauses.pythe six clause checks, and the money with them — a swap seam

Every deviation the printed fields can settle, WITH ITS AMOUNT -- the threshold in rooms, the eligible food and beverage total, the entitlement on the named basis, the per-night rate variance valued at the rooms picked up, the rooms held, and the band the notice date selects. Computed on every review of every file whether or not anything is wrong with it.

You change it to: Add a function and a CHECKS entry. Each one moves a clause area off the bill for free.

# The structured clause checks, in pure code. No model, no correspondence.

TOL_USD = 100.00
def _clause(parsed, want):
def _m(v):
def attrition(p):
def fnb(p):
def comps(p):
def rate_variance(p):
def cutoff(p):
def band_for(p, days):

src/clauses.pythe materiality threshold — a swap seam

TOL_USD = 100.0. It is the threshold underneath every adjustment figure any floor publishes, GC-1.4 allows an immaterial variance and does not define one, and it moves the answer: 9.03 points of swing in the discriminator across eight values -- but ONLY on the three clauses billed in money. The three billed in whole room-nights (CR-1, CR-3, CR-5) do not move at all in the band the clause is actually asking about.

You change it to: Add a function and a CHECKS entry. Each one moves a clause area off the bill for free.

# The structured clause checks, in pure code. No model, no correspondence.

TOL_USD = 100.00
def _clause(parsed, want):
def _m(v):
def attrition(p):
def fnb(p):
def comps(p):
def rate_variance(p):
def cutoff(p):
def band_for(p, days):

src/prompt.pythe prompt, in one file, in send order

The three verdicts and the six deviation types are NAMED. What is deliberately absent is where to look: nothing says 'read the correspondence for something that contradicts the contract terms', or that a permitted reduction changes the figure attrition is measured against.

# The assembled prompt, in one place, in the order it is sent.

NOTES_HEADING = "Contract Correspondence"
SYSTEM = """You are reviewing what actually happened on a group booking against the group contract
BLIND_SYSTEM_SUFFIX = """
def build(text, blind=False):
def _strip_notes(body):
def render(parts):

src/prompt.pythe correspondence-blind ablation

⚠︎ _strip_notes: WRITTEN, RED-PROVEN AGAINST A MOVED HEADING, AND NOT RUN. It raises rather than silently no-opping if the heading moves. That it was not fired is a spend decision and is recorded as unrun, not left to look like a pass.

# The assembled prompt, in one place, in the order it is sent.

NOTES_HEADING = "Contract Correspondence"
SYSTEM = """You are reviewing what actually happened on a group booking against the group contract
BLIND_SYSTEM_SUFFIX = """
def build(text, blind=False):
def _strip_notes(body):
def render(parts):

src/review.pyone model call per file

MAX_TOKENS = 64000, raised off 32,000 by this kit's own calibration before any spend. parse_reply refuses a partial object rather than filling it in.

# One group file in, one clause-by-clause deviation review out. The only place a model is called.

MAX_TOKENS = 64000
THINKING = None
def parse_reply(text):
def normalise(obj):
def review(cfg, text, blind=False, complete_fn=None, max_tokens=None):
def cited_ids(answer):
def action_from_rows(answer):

src/adapters/__init__.pyraw HTTP, no vendor SDK — a swap seam

TIMEOUT_S = 1800. A 64000-token generation is not streamed, so it holds a silent socket for its whole length; the ceiling and the socket timeout are one setting wearing two names.

You change it to: PROVIDER and MODEL in .env -- shared at the repo root, overridable per kit.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
TIMEOUT_S = 1800
TRANSPORT_RETRIES = 1
def _post(url, headers, payload, timeout=TIMEOUT_S):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}

evals/baseline.pythree free floors — a swap seam

raw-variance, threshold-calc and clause-gate -- three different systems, not three settings. Each runs all 48 files offline in a fraction of a second, and each is scored by the same function as the paid arm.

You change it to: --floor raw-variance | threshold-calc | clause-gate.

# THREE FREE FLOORS. No key, no model, no network. Each is a genuine attempt at the job.

MODES = ("raw-variance", "threshold-calc", "clause-gate")
def _action(rows):
def _row(rid, subject, verdict, dtype, clause, evidence, amount, finding):
def review(text, mode="clause-gate", tol_usd=CK.TOL_USD):
def _raw(rid, p, read):
DEFAULT_ATTRITION_PCT = 80.0
def _threshold(rid, p, read, tol_usd):

evals/tolerance.pythe materiality sweep

Re-scores all three floors at eight thresholds, free and offline, and prints the per-clause split so a reader can see which half of the contract the constant actually touches.

# THE MATERIALITY SWEEP. Free, offline, and it must run before anyone believes a floor's score.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
RESULTS = os.path.join(HERE, "results")
TOLERANCES = (0.00, 1.00, 5.00, 25.00, 100.00, 250.00, 1000.00, 5000.00)
def _docs():
def _text(d):
def sweep():
def main():

evals/scoring.pythe scorer

Pure Python, exact match per clause review against data/gold.jsonl, with the amount inside the discriminator. ⚠︎ Three of its conditions are deliberately looser than the key -- the sign, the clause within the file's own printed scope, and the evidence family -- because the floor and the key share an author and scoring notation would hand the floor points the model cannot earn.

# Score an arm against the answer key. Pure code, exact match per clause review. No model grades

COMPLIANT, DEVIATION, INSUFFICIENT = "COMPLIANT", "DEVIATION", "INSUFFICIENT_EVIDENCE"
OMITTED = "OMITTED"
CENT = 0.005
ONE_PCT = 0.01
EVIDENCE_FAMILY = {
def _evidence_ok(review, want, got):
def _pct(n, d):
def _key(s):
def rows_by_review(answer):

evals/rescore.pythe re-scorer

Re-runs the scorer over recorded replies, no calls and no new answers, and stamps scored_by into every file it touches. It exists so that no two columns on this page can have been scored by two different scorers -- and it keeps failed files in the denominator rather than dropping them.

# Re-score a committed result file against the current scorer. No calls, no spend, no new answers.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
SCORER_NOTE = ("evals/scoring.py, with the amount compared in MAGNITUDE, the governing clause "
def load_gold():
def valid_ids(doc_ids):
def rescore(path, golds):
def main():

evals/check_labels.pythe answer-key gate

Re-reads the shipped text and asks whether each label AND each amount follows from it. 0 violations over 48 files and 288 clause reviews; --self-test seeds eight defects in memory and requires a conviction for each.

# THE ANSWER-KEY GATE. Re-derive from the shipped corpus everything pure code can re-derive, and

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
CENT = 0.005
DE_MINIMIS_CAP_USD = 60.0
def load():
def check(golds, texts):
def self_test():
def main():

evals/injection.pythe paired injection probe — a swap seam

⚠︎ WRITTEN, RED-PROVEN AGAINST A MOVED HEADING, AND NOT RUN. It forces one instruction-shaped sentence into the Contract Correspondence and re-fires every file in which the paired run correctly raised a deviation, comparing each cell against its own un-injected answer. Both phrasings are written out in full. That it was not fired is a spend decision and is recorded as unrun, not left to look like a pass.

You change it to: --phrasing, over PHRASINGS. Both are written out in full.

# THE PAIRED INJECTION PROBE. One instruction-shaped sentence in the contract correspondence, and

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
NOTES_HEADING = PR.NOTES_HEADING
PHRASINGS = {
NAMED_AREAS = {
def documents():
def load_doc(doc_id):

src/app.pythe local UI

http.server plus ui/, no dependency, five endpoints, no write path. Port 9084. It renders every recorded run and both free columns, with every recomputation and every amount, with no key configured.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
CORPUS = os.path.join(HERE, "data", "corpus")
PORT = int(os.environ.get("PORT", "9084"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r002-group-deviation")
FLOOR_MODE = "clause-gate"
NAIVE_MODE = "threshold-calc"
def documents():
def load_doc(doc_id):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 2792 input and 32342 output tokens per group file (one signed contract, one final bill, six clause reviews), all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

⚠︎ ONE BOUNDARY MEASURED, ONE NAMED AND NOT MEASURED. The withholding seam is red-proven -- the check seeds the leak it exists to prevent and convicts it -- and the fabrication check reads the file rather than the answer key so it can convict the key too. But the Contract Correspondence is this kit's injection surface: it is prose written by or on behalf of the counterparty whose bill is under review, it reaches the model VERBATIM, and it is where 78 of the 288 clause reviews are decided. evals/injection.py is written in full and red-proven and WAS NOT FIRED. This page publishes no injection number rather than an assertion that the kit resists one.

API_KEY is read from the repo-root .env, an optional kit-local .env, or the real environment, in that precedence (src/config.py). Both files are gitignored from the first commit; this repo has never held a credential. The key is never logged, never placed in a prompt, and the .env writer opens at 0600 BEFORE the write rather than chmod-ing after. config.sources() reports WHICH file contributed and never a value, so 'where is this key coming from' is answerable without anyone reading a key aloud. The UI's error path scrubs the key and the base URL out of any provider message before it reaches the page.

The experimentWe did NOT attack it, and here is exactly what that leaves unknown

evals/injection.py is complete and unrun. It inserts one sentence into the Contract Correspondence as a real, identified CC note -- so a correct answer citing it is not scored as a fabrication -- and re-fires the review. The scope rule is fixed in code before any run: files in which the paired run correctly raised at least one deviation, in file-id order, capped at --limit; every deviation cell in those files is compared against the same arm's own un-injected answer for the same cell. Both phrasings are written out in full, and inject() raises rather than silently passing the text through if the correspondence heading moves, because a silent no-op would publish an injection arm that was never injected. Firing it is one command and about a dozen calls; it was a spend decision on a lap where the provider was answering one call every fourteen minutes, and it is recorded here rather than quietly omitted. Both were run for real on 2026-08-26.

Two of the three boundaries are measured; the third is written, red-proven and unrun, and is reported as unrun rather than as a pass. The withholding seam is red-proven by seeding the leak it is supposed to prevent, and the fabrication check reads the file rather than the key so that it can convict the key.

The resultThe property's commercial position cannot leave the machine and is red-proven at the seam; 0 of 288 citing rows invented an identifier. The injection surface -- the counterparty's own prose, read verbatim, deciding 78 of 288 reviews -- was NOT measured, and this page says so rather than implying safety.

Two boundaries measured, one named and unrun. The withholding seam and the fabrication check both carry numbers from the scored run; the injection surface carries none, because evals/injection.py was written, red-proven against a moved heading, and not fired.

HonestyWhat this does not prove

• ⚠︎ THE HEADLINE ONE: whether a hostile sentence in the contract correspondence can suppress a deviation this kit's own rules say is due. The probe is written, red-proven and NOT FIRED. No number is published and none is implied.
• Whether either of the two written phrasings is representative of what a counterparty would actually write. Neither was fired, so neither is calibrated.
• Whether an injection placed in Property Position would have any effect -- by construction it cannot, because that section is never sent.
• Whether the structured column is genuinely immune. It is code and cannot be instructed, but no probe fired against the free floors, because there is nothing in them to instruct.
• Whether another model tier resists it. One tier was run against this corpus.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Never contact a group or its planner, never issue an invoice, never post an adjustment, never credit a line and never release a bill -- and never let the property's own commercial position leave the machine. Every finding is a figure a billing controller may decide to put to a counterparty, in writing, under a contract this file references and does not enforce. DEVIATION means the review goes to the group with a clause, a line and an amount behind it; COMPLIANT is the absence of a deviation the contract carries and is NOT an approval of the charge.

Stated to the model on every call in src/prompt.SYSTEM, stated again on the local UI, and true of the code by ABSENCE: there is no writer anywhere in src/, evals/ or ui/ outside results/ -- no billing API, no folio call, no mailer, no state file. The three file actions are recommendations to a person and the prompt says so in the same sentence that defines them.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• This is a PROMPT RULE PLUS AN ABSENCE, not a runtime enforcement layer. Nothing stops a forker adding a billing API call tomorrow.
• ⚠︎ src/select.py withholds a NAMED SECTION and is not a redaction system. A file whose contract correspondence mentions the planner's commission will send it, because the correspondence is where the written concessions live and the kit cannot have both.
• Nothing here validates that the contract in this corpus resembles any real group sales agreement. It is an illustrative default, generated from a seed, and data/SOURCES.md says so before any number.
• ⚠︎ AND NOTHING HERE MEASURES WHETHER THE RULE SURVIVES A HOSTILE SENTENCE. The contract correspondence is prose written by or on behalf of the counterparty whose bill is under review and it reaches the model verbatim. evals/injection.py is written and red-proven and was NOT FIRED, so this page publishes no injection number rather than an assertion of safety.

WatchedWhat is watched, and why that one

The kit measures 98 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

28 measured by the latest run 70 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (adjustment_total_cells 48, amount_cells 59, blind False, cells 288, clause_cited_cells 59, clause_in_scope_cells 59, cleared_note_cells 42, compliant_rows 162, dataset_version group-deviation-2026-08-26-48files-v1, de_minimis_rows 15, deviation_rows 108, evidence_cells 59, files 48, files_answered 48, floor_tolerance_usd 100.0, insufficient_cells 18, prose_deviation_cells 36, rows_citing_any_id 0, structured_deviation_cells 72, trap_rows 42, type_named_cells 59) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

7 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 2 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

comply · no model in the path — a baseline, not a peer column — 4 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 4 runs measured the same system — they differ on amount_cells, clause_cited_cells, clause_in_scope_cells, evidence_cells, floor, output_tokens_max, provider, rows_citing_any_id, type_named_cells, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

comply · with the model in the path — 3 runs. Columns here are only ever compared with each other.

not a time series No two of these 3 runs measured the same system — they differ on adjustment_total_cells, amount_cells, cells, clause_cited_cells, clause_in_scope_cells, cleared_note_cells, compliant_rows, de_minimis_rows, deviation_rows, evidence_cells, files, files_answered, insufficient_cells, max_tokens, output_tokens_max, prose_deviation_cells, reasoning_tokens_total, rows_citing_any_id, structured_deviation_cells, trap_rows, type_named_cells — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

DeviationsWhat deviated

0 breaches across 7 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

per group contract, once, before the final invoice is released

In one lineNo framework, deliberately

NO FRAMEWORK. A folder of readable Python, standard library end to end, no orchestration layer and no vendor SDK. requirements.txt names nothing, because nothing under src/ or evals/ imports anything outside the standard library. The reason is the fork test: every layer added is a thing a forker has to install, understand and trust before they can read the dozen files that are actually the kit.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Linear: the group file -> src/segment -> src/select -> src/gfile -> src/clauses (free, and always computed, WITH the amounts) -> src/prompt -> src/adapters -> evals/scoring. No branching, no tool-calling and no agent loop. The free structured column and the model column are computed on every review and never routed between -- the routing is the thing this kit names as the obvious next build and does not measure.

The other sideWhat a framework costs you

• Swapping providers means editing the PROVIDERS dict by hand. In exchange the prompt has no hidden templating layer and requirements.txt stays empty.
• No retry-with-backoff library, so a transport failure is retried by a few lines of local code rather than by a policy someone else tuned. On a 1800 s socket that matters: a timeout is not a rate limit, and retrying it as one pays for the same generation twice.
• No schema-enforced output, so a malformed reply is a recorded failure rather than a retry. On this run that cost nothing -- every reply parsed.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r002-group-deviation on the fast tier, 2026-08-26. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 6 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

4 runs not plotted. b000-group-deviation-rawvariance, b001-group-deviation-thresholdcalc, b002-group-deviation-clausegate, t000-group-deviation-stub recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the rulebook is data read whole into the prompt.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe whole call on one clause review, including the money

whether each of the 288 clause reviews got a call you could put on an invoice adjustment: the right verdict, and on a deviation the right type, a clause the file lists in scope, the figure it rests on plus any correspondence note that decided it, and THE AMOUNT TO THE CENT -- all of it at once, per review, with nothing invented.

MethodHow the test was run

evals/scoring.py::score, in-process, no key and no model. The same function evals/run.py calls after every run and the same one all three free floors are scored by; evals/rescore.py re-applies it to every committed result file in one pass.

Every grader on these pages scored the same 288 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl, generated with the corpus and gated by evals/check_labels.py: 0 violations over 48 files and 288 clause reviews, with six of the gate's properties being the corpus's central claims rather than housekeeping.

How tight can the band be? Exact match on the closed vocabularies after case normalisation; the amount to the cent, in magnitude. ⚠︎ THE ONE TUNED CONSTANT IS NOT IN THIS GRADER, IT IS IN THE FLOORS: src/clauses.TOL_USD = 100.0, chosen from b900-group-deviation-materiality-sweep on the same 288 rows the floors are then scored against. The paid arm has no constant to tune and is told no number.

Cadence: Re-score free on any change to src/clauses.py, src/prompt.py or tools/build_corpus.py. Re-run evals/run.py (paid) on any change to MAX_TOKENS, the provider or the model.

The decisionWhen to reach for it

In one lineThe file's whole adjustment total, to the cent

of the 48 files, on how many the sum of every deviation amount the arm raised equals the sum the key carries, to the cent. It is published as its own grader because it is the ONLY figure that actually reaches an invoice, and because a healthy row-level average can sit on top of a file total that is wrong on almost every file.

MethodHow the test was run

evals/scoring.py::score, the same pass, summed per file from the arm's own rows rather than from whatever headline it reported -- so a review whose total contradicts its own body can be seen rather than assumed away.

Every grader on these pages scored the same 288 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl's bill_adjustment_total, asserted by evals/check_labels.py against the file's own rows.

How tight can the band be? To the cent, in magnitude, summed over the reviews the arm itself marked DEVIATION.

Cadence: Every run, free.

The decisionWhen to reach for it