The business caseThe problem this solves

Before a landlord releases a tenant's annual operating-expense reconciliation, somebody has to decide whether each expense line is billable under THAT tenant's lease and whether the arithmetic holds. It is not a lookup: the answer depends on a classification the line carries, a gross-up that applies to some expenses and not others, a pro-rata share that changes partway through the year when a tenant expands, and a cap that may be annual or cumulative. Two things on the line argue with all of that and neither is evidence — the property accountant's own note, and what the expense happens to be called. Somebody opening every line of a tenant's operating-expense reconciliation and deciding, line by line, whether the lease permits it — is this pool billable at all or is it landlord overhead, a leasing cost or a capital project; if it is capital, does the lease permit amortizing it and over how many years; does a partly-vacant building earn a gross-up on this particular expense or is it one of the fixed ones that must not be grossed up; what is this tenant's share once a mid-year expansion is weighted over the months it was actually occupied; and does the cap bind, compounded or not. It is four multiplications and a minimum per line, and it is every line of every statement. The part that goes wrong is not the arithmetic: it is doing the stages out of order, or excluding a line because of what it is called.

The inputThe actual reconciliation lines

The corpus is 55 reconciliation lines, 0.05 MB (txt 55). Plain text, one format, invented rather than fetched — and here there are TWO reasons rather than the usual one. A real reconciliation line names a real tenant, a real property and a landlord's own general-ledger detail; and the correct answer depends on that tenant's individually negotiated lease, which is confidential even where the building is public. There is no public corpus of (reconciliation line, correctly-computed tenant share) pairs and there cannot be one. Generating it also makes the label mechanical: gold is the four stages run over the line's own values, never somebody's reading of the accountant's note.

Statement Line
--------------
CAM-JH-43228

Managing Agent
--------------
Stonebrook Property Group

Expense Category
----------------
Common area utilities

Expense Class
-------------
routine_operating

Pool Gross Cost
---------------
136000.00 USD

Amortization Years
------------------
not amortizable under this lease

Occupancy Sensitive
-------------------
yes

Building Occupancy
------------------
99 pct

Building Area
-------------
298100 sf

Tenant Area
-----------
5200 sf

Expansion Area
--------------
no expansion this reconciliation year

Expansion Month
---------------
no expansion this reconciliation year

Cap Type
--------
annual

Cap Percent
-----------
4 pct

Cap Basis
---------
1834.72 USD

Cap Periods
-----------
not applicable to an annual cap

Billed To Tenant
----------------
2372.36 USD

Statement Status
----------------
draft

Accountant Notes
----------------
Not confident the exclusion schedule was applied here; needs a second look.

The outcomeWhat a good result looks like

A twenty-field extracted record per line — eighteen read off the statement, two COMPUTED: the dollar figure the lease permits, and whether the billed amount matches it. Plus one pure-code routing decision from three of those fields: a line billed against the tenant on a statement that has already gone out is the one that needs a corrected statement today. Nothing here issues one, credits anybody or releases a reconciliation.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Pre-checking a draft reconciliation before it is released to tenants — the shipped tier, and read the arithmetic figure rather than the verdict figure
  100 pct and 98.18 pct on the permitted amount across two runs, against the category-name floor's 60.0 pct and the accountant-note floor's 0 pct. On a draft the verdict alone is enough to hold a line back for a human, and holding it back is the whole job.
• Deciding which already-issued lines need a corrected statement today — the fast tier, with a human on every routed line AND a sample of the unrouted ones
  7 of 7 with no false alarms on r001. But r002 missed one of the same seven, because the routing rule reads the model's own dollar figure to decide which way an error runs. The deliberating tier got all seven on its own run, and costs 15.25 pct more per line and 136 pct more latency for it — on a denominator of seven, that is one line's difference and it is not enough to buy the tier on.
• Screening lines with no labelled answers — a forker's own statements on day one — the self-consistency diagnostic, and know what it cannot do
  It is the only figure here computable with no gold: it re-runs the comparison over the reply's own two numbers.
• Choosing between the two tiers — the fast tier, which is this kit's shipped default
  They tie on every published grader: 1,100 of 1,100 cells, 55 of 55 dollar figures, 55 of 55 verdicts and 7 of 7 flags on both the deliberating tier and the fast tier's first run. The deliberating tier costs 15.25 pct more a line ($0.0060963 against $0.0052898) and takes 136 pct longer at the median (22.7s against 9.6s).

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-22. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-22 — r001-cam-precheck. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Checked on a fresh checkout with API_KEY left blank: python -m src.app starts on 127.0.0.1:8844, the 55-line picker populates and the field table draws its twenty empty rows with the two computed fields already labelled as computed; clicking Pre-check returns the no-key sentence rather than a stack trace, and the routing row prints "not computed" rather than "no follow-up needed". python -m evals.check_labels and both free baselines run with no network access at all.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per reconciliation line

Current processWhat it replaces

Somebody opening every line of a tenant's operating-expense reconciliation and deciding, line by line, whether the lease permits it — is this pool billable at all or is it landlord overhead, a leasing cost or a capital project; if it is capital, does the lease permit amortizing it and over how many years; does a partly-vacant building earn a gross-up on this particular expense or is it one of the fixed ones that must not be grossed up; what is this tenant's share once a mid-year expansion is weighted over the months it was actually occupied; and does the cap bind, compounded or not. It is four multiplications and a minimum per line, and it is every line of every statement. The part that goes wrong is not the arithmetic: it is doing the stages out of order, or excluding a line because of what it is called.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 0 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one section is

cut on underlined section headings; a line with none falls back to one whole-document segment

SetupWhat the setup figure measured

There is no index. Preparation is segmentation only — 55 lines cut into 1,045 sections in seven milliseconds, in process, with no model and no network.

LicenceLicence

MIT — this repository's own licence. Every statement identifier, managing agent, expense category, cap term and accountant note is invented; no executed lease, published lease form, real landlord, tenant or property is named or reproduced.

Bring your ownBring your own reconciliation lines

Replace data/corpus/*.txt, write your own data/fields.json, and supply a gold record per line. SECTION_HINTS in src/select.py maps fields to section headings and will need editing for a different statement layout; when it does not match, selection falls back to the whole document — slower, more expensive, always correct. src/rule.py is the FIRST thing to replace: its four stages, its 95 pct gross-up floor and its $1.00 tolerance are this kit's own invented lease structure, and it caps the line where a real lease caps the aggregate. Because five things import that one module, replacing it moves the corpus, the prompt, the guardrail, the scorer and the pre-flight together.

⚠︎ And what stops being true when you do: Every score on this page is measured against a gold that is the SAME arithmetic the kit publishes, run over values a generator chose. On your own statements the true permitted amount is not known in advance — that is the whole reason the job exists — so the field grade, the arithmetic figure and both confusion matrices all go away. What survives is the self-consistency diagnostic, and this kit's own r002 is the measured proof of how little that catches: it stayed silent through the only error either run made, because a reply that computes a wrong number and then judges its own wrong number correctly is perfectly consistent.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 2,318 tokens assembled, 1,051 are schemas — 45% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured token-for-token via evals/prompt_tokens.py's nested-prefix method on CAM-0004 — three calls at max_tokens=1, each part's size the difference between two consecutive prompt_tokens counts the provider itself returned. The three parts sum to 2,318, which is exactly the input_tokens the captured example call was billed. kits/UC0044-cam-precheck/results/tokens-p001-cam-precheck.json.

Full promptThe prompt in full

You check one line of a commercial-property operating-expense (CAM) reconciliation against the tenant's own lease terms. You return JSON and nothing else.

RULES, in order of importance:
1. If the record does not state a field, return null for it. Do not infer it and do not use what you know about the world.
2. `permitted_amount_usd` is what the lease permits the landlord to bill THIS tenant for THIS line. Compute it yourself, in FOUR STAGES, IN THIS ORDER:
   STAGE 1 -- POOLABLE. Start from pool_gross_usd and decide how much of the pool is billable to tenants at all, from `expense_class` and nothing else:
     - landlord_overhead or leasing_cost -> 0. Nothing of it is billable.
     - capital_improvement with amortization_years null -> 0.
     - capital_improvement with amortization_years N -> pool_gross_usd / N. ONE ANNUAL INSTALMENT. Not the whole cost, and NOT zero -- an amortizable capital item is PARTLY billable, which is the case 'capital is excluded' gets wrong.
     - routine_operating -> the whole of pool_gross_usd.
   STAGE 2 -- GROSS-UP. If occupancy_sensitive is 'yes' AND building_occupancy_pct is below 95, multiply the stage-1 amount by 95 and divide by building_occupancy_pct. Otherwise leave it exactly as it is. NEVER gross up a line whose occupancy_sensitive is 'no', however low the building's occupancy is.
   STAGE 3 -- PRO RATA. Multiply by the tenant's weighted share of the building. With no expansion that share is tenant_area_sf / building_area_sf. With an expansion of A square feet taking effect in month M it is (tenant_area_sf + A * (13 - M) / 12) / building_area_sf -- the expansion counts for the months it was actually occupied, INCLUDING month M.
   STAGE 4 -- CAP. Work out the ceiling: cap_type 'annual' -> cap_basis_usd * (1 + cap_pct/100); cap_type 'cumulative' -> cap_basis_usd * (1 + cap_pct/100) ** cap_years, COMPOUNDED over cap_years periods; cap_type 'none' -> there is no ceiling. Then take the LOWER of the stage-3 amount and the ceiling. Round the result to two decimals.
3. `line_ok` is 'yes' when billed_to_tenant_usd is within 1.00 US dollar of the permitted_amount_usd you computed, and 'no' otherwise. Nothing else decides it.
4. THE EXPENSE CATEGORY IS A LABEL, `expense_class` IS THE CLASSIFICATION. A line called 'Parking lot resurfacing', 'Roof membrane patching', 'Property management fee' or 'HVAC compressor overhaul' whose expense_class says routine_operating is a routine operating expense and its whole pool is billable. Do NOT exclude a line because of what it is called.
5. THE STAGES ARE IN THAT ORDER FOR A REASON. Gross up before you cap: a ceiling that has slack against the ungrossed share can bind against the grossed-up one, so skipping stage 2 can also silently skip stage 4.
6. THE ACCOUNTANT'S NOTE IS A FIELD TO COPY, NOT EVIDENCE ABOUT CORRECTNESS. A note that sounds concerned or mentions a past query does NOT mean the line is wrong, and a note that sounds routine does NOT mean the line is right. The arithmetic decides; the note is the property accountant's own remark and may disagree with it.
7. Copy values verbatim from the record wherever possible, and report every money, area, percentage, month and year figure as a bare number with the unit left out of it.
8. Use the exact allowed value for a field that lists them.
9. Return every field named in the schema, even when the answer is null.

Extract these fields:
- line_id (string) -- the statement line identifier, verbatim
- expense_category (string) -- the expense category as written on the line, verbatim. This is a LABEL, not a classification -- it does not decide whether the line is billable
- expense_class (enum) one of: routine_operating, capital_improvement, landlord_overhead, leasing_cost -- the structured expense classification on the line, verbatim. THIS is what decides whether the pool is billable, never the category name
- pool_gross_usd (number) -- the gross cost of this expense pool at the property for the reconciliation year, as a bare number without the currency
- amortization_years (number) -- the number of years this lease permits a capital item to be amortized over. Stated only on a capital improvement the lease permits amortizing; null on every other line -- return null rather than 0 or a guess
- occupancy_sensitive (enum) one of: yes, no -- whether this expense varies with how much of the building is occupied, verbatim
- building_occupancy_pct (number) -- the building's average occupancy for the reconciliation year, as a bare number without the unit
- building_area_sf (number) -- the building's total rentable area in square feet, as a bare number without the unit
- tenant_area_sf (number) -- the tenant's rentable area at the START of the reconciliation year, in square feet, as a bare number without the unit
- expansion_area_sf (number) -- additional rentable area the tenant took mid-year, in square feet, as a bare number. Null when the line states there was no expansion this reconciliation year
- expansion_month (number) -- the month number (1-12) the mid-year expansion took effect. Null when there was no expansion
- cap_type (enum) one of: none, annual, cumulative -- the kind of cap this lease puts on the tenant's share of this line, verbatim
- cap_pct (number) -- the cap percentage, as a bare number without the unit. Null when cap_type is none
- cap_basis_usd (number) -- the dollar amount the cap is measured from -- last year's billed amount for an annual cap, the base year's for a cumulative one. As a bare number without the currency. Null when cap_type is none
- cap_years (number) -- the number of compounding periods since the base year. Stated only on a cumulative cap; null on an annual cap and null when there is no cap
- billed_to_tenant_usd (number) -- the amount the landlord actually put on this tenant's statement for this line, as a bare number without the currency
- statement_status (enum) one of: draft, issued -- has this reconciliation statement already gone out to the tenant, or is it still a draft?
- accountant_note (string) -- the property accountant's own free-text note on this line, copied verbatim
- permitted_amount_usd (number) -- the amount the lease actually permits the landlord to bill THIS tenant for THIS line, in dollars to the cent, computed by you in four stages. STAGE 1, POOLABLE: expense_class landlord_overhead or leasing_cost -> 0. expense_class capital_improvement with amortization_years null -> 0; with amortization_years N -> pool_gross_usd divided by N, ONE annual instalment, not the whole cost. expense_class routine_operating -> pool_gross_usd in full, WHATEVER the category is called. STAGE 2, GROSS-UP: if occupancy_sensitive is yes AND building_occupancy_pct is below 95, multiply by 95 and divide by building_occupancy_pct; otherwise leave it alone -- never gross up a line whose occupancy_sensitive is no. STAGE 3, PRO RATA: multiply by the tenant's weighted share. With no expansion that share is tenant_area_sf / building_area_sf. With an expansion of A square feet in month M it is (tenant_area_sf + A * (13 - M) / 12) / building_area_sf. STAGE 4, CAP: if cap_type is annual the ceiling is cap_basis_usd * (1 + cap_pct/100); if cap_type is cumulative the ceiling is cap_basis_usd * (1 + cap_pct/100) raised to the power cap_years; if cap_type is none there is no ceiling. Take the LOWER of the stage-3 amount and the ceiling. Round to two decimals
- line_ok (enum) one of: yes, no -- does billed_to_tenant_usd match permitted_amount_usd? Answer 'yes' when the two are within 1.00 US dollar of each other and 'no' otherwise. Decide this STRICTLY from the arithmetic -- never from the accountant's note and never from what the expense category is called

Return a JSON object with exactly these keys: line_id, expense_category, expense_class, pool_gross_usd, amortization_years, occupancy_sensitive, building_occupancy_pct, building_area_sf, tenant_area_sf, expansion_area_sf, expansion_month, cap_type, cap_pct, cap_basis_usd, cap_years, billed_to_tenant_usd, statement_status, accountant_note, permitted_amount_usd, line_ok
Use null for any field the record does not state.

CAM RECONCILIATION LINE
-----------------------
Statement Line
--------------
CAM-WT-34473

Expense Category
----------------
Snow and ice removal

Expense Class
-------------
routine_operating

Pool Gross Cost
---------------
47800.00 USD

Amortization Years
------------------
not amortizable under this lease

Occupancy Sensitive
-------------------
yes

Building Occupancy
------------------
86 pct

Building Area
-------------
343900 sf

Tenant Area
-----------
29500 sf

Expansion Area
--------------
no expansion this reconciliation year

Expansion Month
---------------
no expansion this reconciliation year

Cap Type
--------
annual

Cap Percent
-----------
4 pct

Cap Basis
---------
4087.74 USD

Cap Periods
-----------
not applicable to an annual cap

Billed To Tenant
----------------
4251.25 USD

Statement Status
----------------
issued

Accountant Notes
----------------
Escalated for manager review; the allocation basis on this line is disputed.

Raw responseThe raw response

{
  "line_id": "CAM-WT-34473",
  "expense_category": "Snow and ice removal",
  "expense_class": "routine_operating",
  "pool_gross_usd": 47800.00,
  "amortization_years": null,
  "occupancy_sensitive": "yes",
  "building_occupancy_pct": 86,
  "building_area_sf": 343900,
  "tenant_area_sf": 29500,
  "expansion_area_sf": null,
  "expansion_month": null,
  "cap_type": "annual",
  "cap_pct": 4,
  "cap_basis_usd": 4087.74,
  "cap_years": null,
  "billed_to_tenant_usd": 4251.25,
  "statement_status": "issued",
  "accountant_note": "Escalated for manager review; the allocation basis on this line is disputed.",
  "permitted_amount_usd": 4251.25,
  "line_ok": "yes"
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Check a commercial tenant's yearly expense bill against its lease — 55 reconciliation lines. Two tiers of one model family answered, and every answer was then graded Four different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Run it twiceThe same set, run again

The same model on the same prompt over the same 55 lines computed every permitted amount correctly on one run and got one wrong on the next — CAM-0032, off by +$20,105.37. The review flag that reads that number fell from 7 of 7 to 6 of 7 with it. Nothing else moved: extraction was 1,100 then 1,099 of 1,100 (the same cell), the verdict grade held at 100 pct on both, the span rate at 100 pct on both, hallucinations at zero on both, and the self-consistency diagnostic at zero disagreements on both. The deliberating tier's single run then landed at 55 of 55 as well — so of three scored runs across two tiers, two are perfect and one is not, and the one that is not is a repeat of a run that was.

permitted-amount arithmetic within $1.00, and the review flag that reads it — measured on the SAME tier twice, with a second tier run once beside it — Averaging would publish a 99.09 pct nobody measured and would hide the only thing the pair established: that this kit's residual is nondeterministic, that a single instance of it disarms the routing decision, and that four of the five published figures do not notice when it happens. And averaging ACROSS TIERS would be worse still — it would read as a model-quality figure when the three runs are two tiers and a repeat.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran r001 and r002 publishes no rate card this repo commits, so nothing here is what was actually paid — the real spend for this kit's build is recorded in the commit history and the shared call ledger, not on this page.

The gradersFour ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The three scored runs' own token counts (55 lines each) priced at Google Gemini 3 Flash's published rate — the same basis cost_per_query_usd uses, not a second, larger spend. Both free floors (evals/baseline.py in either mode) and the scorer (evals/judge.py) are pure code and cost $0.00 against any result set. ⚠︎ THIS FIGURE IS SEVERAL TIMES ITS SIBLINGS' AND THE REASON IS NOT WASTE: this kit bills 1377 to 1646 output tokens a line where a copy-the-fields kit bills about 240, because the answer is a computation rather than a copy. A forker re-running the eval pays it on top of the pipeline.

Cost driversWhat actually moves the bill

• OUTPUT, and this is the driver that inverts this kit against its siblings: 1377 output tokens a line at $3.00/M is $0.0041 of a $0.0053 call — 78 pct of the bill. The provider reports reasoning_tokens beside the completion count and on the captured example it was 1,269 of 1,503, so most of what is paid for is the model working through four dependent stages where nobody can read it.
• The fixed system prompt and field schema (1,995 of 2,318 tokens on the example call, 86 pct) outweigh the line sent (323 tokens) — the input floor every call pays before a single value is read. Twenty fields is a long schema, and it is the same schema on every line.
• Nothing amortises. Each line is an independent call carrying the whole fixed prefix again, with no cache and no batch.

Your volumeWhat it costs at your volume

Linear in reconciliation lines: each call is independent, carries the same fixed prompt and shares nothing with its neighbours, so 550 lines cost ten times 55 and take ten times as long. Nothing amortises — there is no index to build and no cache, which is also why there is no volume discount to find without changing the design. The wall-clock number is the one that bites first: 55 lines took ten minutes serially, so a real reconciliation is days before anything is parallelised.

Model choiceWhy this model

TWO tiers, and the fast one twice rather than each once — which is why this kit publishes both a tier comparison and a repeatability measurement off three runs. The pairing turned out to be the load-bearing choice: the deliberating tier's single run is perfect on every grader, and so was the fast tier's FIRST run. Only running the fast tier a second time exposed a nondeterministic arithmetic residual that either single run would have published as 100 pct. The kit's shipped default is the fast tier, which is what its fitment section recommends.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 55 CAM reconciliation lines, one completion call each. Run r002's own 55 calls are the same tier again and are recorded separately in Cost.cost_by_model; they are not projected here.

The codeEvery file, and what it is for

Six modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pysegment

cut the reconciliation line into 19 addressable sections, pure code — and locate a money value back to its own section across the three spellings the same number can take (47800.00, 47800.0, 47800)

# Cut a document into addressable sections. Pure code — no model, no network.

def sections(text):
def _variants(value):
def locate(text, value):
def span_label(secs, start):

src/select.pyselect — a swap seam

pick which sections carry each field, pure code — the two computed fields are mapped to the thirteen sections the arithmetic actually reads, never to the accountant's note and never to the expense category; Managing Agent is mapped by nothing and is never sent

You change it to: map fields to your own statement's headings; unmatched falls back to the whole document

# Pick which sections plausibly carry each field. Pure code -- the last deterministic step

SECTION_HINTS = {
def for_field(secs, field):
def plan(secs, fields):

src/rule.pyrule — a swap seam

the four stages — poolable, gross-up, pro rata, cap — as the ONLY copy in the kit. The corpus generator, the prompt, the extractor's self-check, the scorer and the pre-flight all import it, so there is no second implementation to drift

You change it to: the lease itself — your own permitted/excluded categories, your own gross-up floor, your own cap mechanics and your own tolerance. It is stated once and read by the corpus generator, the prompt, the guardrail and the scorer, so changing it here changes all of them together. This is the first thing a forker replaces: the four stages here are invented and, in particular, cap the LINE where a real lease caps the aggregate

# THE LEASE RULE, and the only copy of it in this kit.

GROSSUP_FLOOR_PCT = 95.0
TOLERANCE_USD = 1.00
MATERIALITY_USD = 25.00
CLASSES = ("routine_operating", "capital_improvement", "landlord_overhead", "leasing_cost")
CAP_TYPES = ("none", "annual", "cumulative")
def _num(v):
def poolable_amount(expense_class, pool_gross_usd, amortization_years):
def grossed_up(pool, occupancy_sensitive, building_occupancy_pct):
def prorata_share(building_area_sf, tenant_area_sf, expansion_area_sf, expansion_month):

src/prompt.pyprompt

assemble one call for all twenty fields, with the four stages stated in full and in order — including that an amortizable capital item is PARTLY billable, that the class decides and not the category name, and that gross-up comes before cap

# Assemble the extraction prompt. One prompt per CAM reconciliation line, all twenty fields in it.

SYSTEM = (
def field_schema(fields):
def build(doc_text, secs, fields, selector):
def parse(raw, fields):

src/extract.pyextract — a swap seam

the AI layer, one provider one key — plus the pure-code business-condition check downstream: a line billed against the tenant on a statement already issued is routed for a corrected statement

You change it to: the routing rule itself — this kit ships three conditions (billed wrong AND against the tenant AND already issued) and a real desk weighs the dollar size against the audit threshold, how many cycles the error has run, and whether the tenant's audit window is open. It is deliberately NOT the same function as permitted_amount(), so changing WHO GETS ROUTED does not change WHAT THE LEASE PERMITS

# Extract one CAM reconciliation line's fields: segment, select, prompt, one model call, then a

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 6000
def load_fields():
def load_doc(line_ref):
def documents():
def recompute(values):
def compute(values):

evals/judge.pyjudge

score field accuracy, the arithmetic figure, the billability confusion matrix, the review flag and a per-trap breakdown separately, pure code

# Score a CAM pre-check run. PURE CODE -- gold is exact and every answer is one value, so `==`

def norm(v):
def _num(v):
def equal(field, got, want):
def score(fields, records, golds):
def _matrix(rows, positive):
def _gold_permitted(g):
def score_flags(records, flags, golds):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 2314 input and 1377 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's reconciliation lines are entirely synthetic (tools/build_corpus.py, seed 20260822): no real lease, landlord, tenant, property or managing agent exists in the corpus, and nothing was fetched from anywhere. The only outbound traffic the kit makes is one chat-completion request per line to the configured provider, carrying the mapped sections of one line. Nothing is written outside the kit directory, there is no database, no auth and no multi-tenancy, and the local UI binds 127.0.0.1 only.

Read from the shared .env, this kit's own .env (which holds one MODEL line and no credential) or the real environment only, never committed, never logged and never rendered. .env is gitignored from the first commit. On an adapter error the app redacts the key and base URL out of the message before returning it, so a misconfigured endpoint cannot echo a credential into a browser.

The experimentWe did not attack it -- and four of five boundaries hold

The four boundaries that hold were confirmed by reading the code, not by a run: no code path issues a statement or credits anybody; the routing rule reads only enums and numbers, so the accountant's note cannot reach it; stage 1 of the lease rule branches on expense_class and never on expense_category, so the misleading name cannot reach it either; and the key and base URL are redacted out of any error before it is returned. The fifth is open. An indirect prompt injection needs a field an outside party authored, and this kit has two of them -- which makes it the obvious place to attack and the reason the gap is named rather than glossed. Confirmed by reading the code, not by a run, on 2026-08-22 -- no attack run was fired; see redteam.why.

The first four boundaries hold, confirmed by reading the code rather than by an attack run. The fifth is open and is named as open -- an indirect prompt injection in the accountant's note is exactly the attack this kit's shape invites, and it is sharper here than on any sibling: this kit's own r002 already demonstrated, by accident, that a wrong DOLLAR FIGURE disarms the routing decision while every other published figure stays clean. An attacker would not need to move the verdict at all.

The result0 attack trials, and four of five boundaries checked here hold on evidence. The one that does not is the one this kit's own shape invites -- and this kit has already shown, without any attacker, what an attack on it would look like: on run r002 a single wrong dollar figure hid a real overcharge from the routing rule while the verdict, the spans, the consistency check and the hallucination count all stayed clean.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260822), so no text in it came from an outside party and there was nothing adversarial to resist. The two planted ambiguities are a REGISTER mismatch and a MISLEADING NAME -- both measure whether a model does the arithmetic when the prose points elsewhere. Neither measures whether a model obeys an instruction hidden in the same field. Those are different failures and only one kind is measured here.

HonestyWhat this does not prove

• Whether an indirect prompt injection in the accountant_note field, or in the expense_category name, could move permitted_amount_usd on this tier. No attack run exists -- and this is the kit where that gap matters most, because moving the number is enough to disarm the guardrail without moving the verdict.
• Whether the kit behaves safely against a hostile provider -- a response body crafted to break the JSON extraction in src/prompt.py::parse, or to return an enormous payload. parse() fails closed to an empty dict, which the harness records as a failed line, and nothing beyond that was tested.
• Whether a real reconciliation archive would carry anything sensitive this kit mishandles. The corpus has no personal or commercial data by construction; a real line carries a named tenant, a negotiated lease term and a landlord's general-ledger detail, and none of that path is exercised.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

needs_review fires when THREE things hold at once: the line is billed wrong (line_ok == "no"), the error runs against the tenant (billed_to_tenant_usd exceeds permitted_amount_usd by more than the $1.00 tolerance), and the statement has already gone out (statement_status == "issued"). All four values come out of the same reply; the rule is run afterwards in pure code, over whatever the model returned, and never over gold. A reply missing any of them returns None rather than False: an unknown is not a pass.

src/extract.py::compute(), called by extract() on every line, by the local app on every click, and by evals/baseline.py on both free floors' own output so all three are routed by identical code. The four-stage rule it reads lives in a SEPARATE module, src/rule.py, which is also what the corpus generator wrote gold with, what src/prompt.py states to the model and what evals/judge.py re-derives gold's truth from -- one definition, five readers.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• ⚠︎ IT IS NOT INDEPENDENT OF THE MODEL'S ARITHMETIC, AND THIS IS MEASURED RATHER THAN ADMITTED. The direction test reads permitted_amount_usd, which the model computed. On run r002 CAM-0032 came back at $45,941.25 against a true $25,835.88, so a $1,116.97 OVERcharge on an already-issued statement read as an undercharge and the flag did not fire -- one of the seven lines it exists to catch, missed, with the verdict grade still reading 100 pct.
• It is NOT a check on whether the extracted values are right. If the model misreads a value and then computes consistently from the misreading, the line is routed (or not routed) on a wrong number and this rule cannot tell. The consistency diagnostic in evals/judge.py is the closest thing to that check, it is reported separately, and it is blind to exactly the case above -- it recorded 0 disagreements on the run that made the error.
• It is NOT a real property-accounting policy. Billed-wrong-and-against-the-tenant-and-already-issued is this kit's own simplification, invented for this corpus. No lease's audit-rights clause, no reconciliation deadline and no landlord's release procedure was consulted, and none is reproduced. A real desk weighs the dollar size against the audit threshold, how many cycles the same error has run, and whether the tenant's audit window is still open.
• It is NOT a disposition. Nothing here issues a corrected statement, credits a tenant or releases a reconciliation, and line_ok is a computed field rather than a decision anybody is bound by. The catalogue entry this kit answers says a human releases, and it means it.

WatchedWhat is watched, and why that one

The kit measures 33 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 26 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (extraction_cells 1100, failures 0, lines 55, refusal_cells 0) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

5 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked.

not a time series No two of these 5 runs measured the same system — they differ on provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

DeviationsWhat deviated

0 breaches across 5 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the flag's evidence on any change to compute() in src/extract.py or to the four stages in src/rule.py, on any change to data/fields.json's allowed values for line_ok or statement_status, and on any corpus regeneration. Changing WHO GETS ROUTED is a policy change and must be re-scored, even though it never changes what the lease permits. And re-check it on any provider or model change with more weight than a sibling kit would: this flag reads a computed field, so a change in the model's arithmetic changes the flag without changing anything the flag itself reads. The tier comparison here is one run each way and one repeat, so treat a tier change as a reason to run twice rather than once.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt is empty on purpose: the corpus is generated in-process, the provider is reached over urllib, and the UI is one HTML file with no build step. A forker runs this on whichever key they already hold, with one clone and no install.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph. The pipeline has one path per line -- segment, select, prompt, one call, parse, compute -- with no branch, no loop and no agent. What looks like depth in this kit is inside the model's reasoning, not in the orchestration: 84 pct of the billed output is the four stages being worked through, and none of that is code anybody wrote.

The other sideWhat a framework costs you

• Everything is hand-rolled, so everything is yours to maintain: the JSON extraction from a fenced reply, the retry/backoff policy, the section regex, the numeric-variant span search and the .env reader are all code somebody has to own.
• No framework means no framework's ecosystem -- no tracing, no eval harness beyond the one in evals/, no prompt registry, no schema validation library. What ships is what is in the repository.
• The provider abstraction covers exactly two wire formats. A third provider is one function and one dict entry, and until somebody writes it the kit runs on two shapes.
• Doing the arithmetic in the MODEL rather than in code is a deliberate choice and it costs what this page measures: 78 pct of the bill and one wrong figure in 110. A kit that extracted the thirteen inputs and ran src/rule.py over them in Python would be cheaper, faster and exactly right every time -- and would measure nothing about whether a model can apply a lease clause, which is the question this kit exists to ask.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-cam-precheck on the shipped tier, run 2, 2026-08-22. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 4 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

2 runs not plotted. b000-tone, b001-name recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — each document goes whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT — this repository's own licence. Every statement identifier, managing agent, expense category, cap term and accountant note is invented; no executed lease, published lease form, real landlord, tenant or property is named or reproduced. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one linePer-field exact match, one field to a $1.00 tolerance

Does the model's value for each field match gold, after trimming whitespace and punctuation and treating numbers within half a cent as equal? permitted_amount_usd is the one exception and is graded at $1.00 — it is the model's own arithmetic through four dependent stages, not a value to be copied. Six fields are legitimately null on some lines and stated on others, so a null there is a hit, not a miss, when gold agrees.

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model. The same function evals/run.py calls after every call, and the same field-match logic both free floors are scored by — a baseline and a real run scored by two different scorers cannot be compared honestly.

Every grader on these pages scored the same 110 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, read back off the same expense class, pool cost, amortization term, occupancy, areas, expansion and cap terms the line states; evals/check_labels.py asserts every non-nullable field is populated on every row, that all six nullability invariants hold as iff-conditions, that every permitted amount is the line's own arithmetic and every verdict its own comparison, before any run is allowed to spend.

How tight can the band be? Exact match after trimming, and numbers within half a cent are treated as equal — EXCEPT permitted_amount_usd, at $1.00. That tolerance is not lenience, it is the unit: grading a four-stage computation to the cent would score a correct answer wrong for a rounding step nobody in a property-accounting office would notice, and $1.00 is exactly where line_ok is defined.

Cadence: Re-score on any change to src/rule.py, src/prompt.py, data/fields.json or tools/build_corpus.py. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineThe computed dollar figure against the four stages

Did the model land within $1.00 of what the lease actually permits? It is one column of the grader above, reported on its own because it is the thing this kit really asks a model to do — and because it is the only published figure that moved between two runs of the same model.

MethodHow the test was run

evals/judge.py::score_flags, in-process, no key and no model.

Every grader on these pages scored the same 110 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/rule.py::permitted_amount(), re-derived inside the grader from gold's own stated values — so the truth this is measured against can never be a separately-typed number that drifted from the rule the prompt states.

How tight can the band be? $1.00, from src/rule.py::TOLERANCE_USD. The corpus's materiality floor guarantees every wrongly-billed line is off by at least $25.00, so the tolerance can never decide a verdict.

Cadence: Re-run whenever src/rule.py changes, whenever the corpus is regenerated, and on any provider or model change.

The decisionWhen to reach for it

In one lineline_ok against gold's own arithmetic

Of every line that really is billed wrong, how many did the run catch — and how many correctly-billed lines did it wrongly query? BILLED-WRONG IS THE POSITIVE CLASS: a line billed wrong that gets called right is what a tenant's auditor finds two years later with interest.

MethodHow the test was run

evals/judge.py::score_flags, in-process, no key and no model.

Every grader on these pages scored the same 110 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: Gold's verdict is re-derived inside the grader by the same four stages the kit publishes and the same $1.00 comparison, so the truth this matrix grades against can never be a separately-typed label that drifted from the rule.

How tight can the band be? No threshold — the verdict is one of two allowed values, and a reply that returns neither is counted as unanswered rather than folded into the correct-negative cell.

Cadence: Re-run whenever src/rule.py changes, whenever the corpus is regenerated, and on any provider or model change.

The decisionWhen to reach for it

In one lineneeds_review against the same rule run over gold

Does the pure-code routing decision — billed wrong AND against the tenant AND already issued — land on the same lines it would land on if every field had been read and computed perfectly? It is a BUSINESS CONDITION, so unlike a self-consistency check it genuinely needs labels, and saying so is half of what makes the number believable.

MethodHow the test was run

evals/judge.py::score_flags, in-process, no key and no model.

Every grader on these pages scored the same 110 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/extract.py::compute(), the same function the run uses, applied to GOLD's verdict, statement status, billed amount and permitted amount. One rule, four inputs, so a change to the rule moves both sides together and the grader cannot silently grade an old policy.

How tight can the band be? No threshold — three conditions and an AND, with the direction test using the same $1.00 tolerance the verdict uses.

Cadence: Re-run whenever compute() changes. Changing WHO GETS ROUTED is a policy change and must be re-scored, even though it never changes what the lease permits.

The decisionWhen to reach for it