The business caseThe problem this solves

Before a lease can be administered it has to be abstracted: rent, term, options and clauses lifted out of a document pack into an eighteen-field record that a portfolio runs on for years. An abstract entry is only useful if somebody can check it, and checking it means turning to the page. A drafted value with no traceable citation is not a weak entry — it is not an entry at all, because confirming it costs the same as drafting it did. Somebody opening a lease document pack and typing an abstract from it — the eighteen terms a portfolio is administered from for a decade — and, for each one, recording WHERE in the pack it is stated so that the administrator whose name is on the abstract can confirm it against a page instead of re-reading the whole document. The part that goes wrong is not the reading. It is the SOURCE: every pack opens with a Basic Lease Information summary sheet that restates eight of the eighteen terms and says on its own face that it is not part of the Lease, a rent schedule at the back that repeats what an Article grants, and — on 12 of these 50 packs — an amendment that is a different row's job entirely. A term read off any of those can be perfectly right and cite a page that binds nobody.

The inputThe actual lease packs

The corpus is 50 lease packs, 0.41 MB (txt 50). Plain text, one format, invented rather than fetched — and here there are TWO reasons rather than the usual one. A commercial lease is a confidential document between two named parties and the negotiated half of it is the half neither party publishes, so there is no public corpus of (lease pack, correctly-cited abstract) pairs and there cannot be one. And generating it is what makes the PAGE mechanical: the generator lays each pack out, paginates it, then LOCATES each governing sentence in the finished text to learn which page it landed on. A page number nobody chose cannot be a page number somebody chose wrongly — and a citation grade whose gold was typed by the person writing the kit would measure nothing at all.

SYNTHETIC LEASE DOCUMENT PACK -- generated for an AI Foundry evaluation kit. Not a real lease.
No real landlord, tenant, property, broker or lease administrator is named. MIT licensed.
File reference LSE-0001.

=== PAGE 1 ===
BASIC LEASE INFORMATION
-----------------------

This summary is prepared for the convenience of the parties and their brokers. It is not
part of the Lease, it does not amend or qualify any provision of the Lease, and in the
event of any inconsistency the provisions of the Lease govern.

    Tenant:                Voss Ridge Advisory Partners LLC
    Building:              Calloway Yards
    Premises:              Suite 1617
    Rentable Area:         41,750 rentable square feet
    Commencement Date:     August 4, 2020
    Expiration Date:       August 3, 2023
    Term:                  36 months
    Monthly Base Rent:     $80,890.62 per month, first Lease Year
    Security Deposit:      $80,890.62

Prepared by the leasing agent. Not to be relied upon as a statement of the terms of the
Lease.

=== PAGE 2 ===
RECITALS
--------

THIS LEASE AGREEMENT is made between the Landlord of Calloway Yards and the Tenant named
in Article 1 below. Landlord is the owner of the Building. Tenant desires to lease the
Premises from Landlord on the terms set out in this Lease. The parties therefore agree
as follows.

Capitalised terms used in this Lease have the meanings given to them in the Article in
which they first appear. The Basic Lease Information sheet bound at the front of this
Lease is a summary only.

Landlord and Tenant have negotiated the terms of this Lease at arm's length, each with
the benefit of its own advisers, and neither party is to be treated as the draftsman of
it for the purposes of any rule of construction.

The outcomeWhat a good result looks like

An eighteen-field abstract per pack, each field a value AND the page it is stated on, plus a pure-code interlock: every field is routed to the administrator of record named on the execution page, and the abstract is commit-READY only when every drafted field carries a citation the pack itself confirms. Nothing here writes to a lease register, and there is no interface to one.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Drafting a first-pass abstract for a lease administrator to confirm — the shipped tier and the full prompt, with the administrator confirming every field
  838 of 838 page citations correct on 50 packs, 95.33 pct value accuracy exact and 100.00 pct once a free-text span boundary is allowed, with no invented terms at all (1.00 absence precision). The confirmation step is not a caveat here, it is the product: every field routes to the named administrator and nothing is committed.
• Checking an existing abstract's citations, with no labels at all — the two no-gold checks together — does the cited page state the value, AND does that page govern
  Both are pure code, need no key and need no gold, and the second one catches the whole summary-sheet and amendment class outright because those pages announce what they are in their own headings. On the free summary floor the first check reads 400 of 400 and the second reads 0 of 400.

And where nothing here is good enough:

• Deciding whether an abstraction pack needs a model at all — neither, until you have measured your own packs — and start with the free last-mention floor
  On THIS corpus a 20-line regex floor reaches 97.67 pct value accuracy and 83.53 pct citation accuracy for nothing, because the corpus is generated from a fixed set of clause templates and the regexes are tuned to them. That figure is a fact about a template, not about lease abstraction.
• Committing an abstract straight into a lease register — not this kit, and not any configuration of it
  11 of 50 abstracts were correct end to end on the best run measured here. There is also no write interface to a register — the catalogue row records that as an open item — and no code path in this kit performs a write of any kind.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-23. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-23 — r001-lease-abstract. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Checked on a fresh checkout with API_KEY left blank: python -m src.app starts on 127.0.0.1:9006, the 50-pack picker populates, the pack's page index draws with the non-governing pages already marked, and the field table draws its 18 empty rows; clicking Abstract returns the no-key sentence rather than a stack trace, and the interlock banner prints — rather than "commit-ready". python -m evals.check_labels, both free baselines, the stub run and python -m evals.rescore all run with no network access at all.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per lease document pack

Current processWhat it replaces

Somebody opening a lease document pack and typing an abstract from it — the eighteen terms a portfolio is administered from for a decade — and, for each one, recording WHERE in the pack it is stated so that the administrator whose name is on the abstract can confirm it against a page instead of re-reading the whole document. The part that goes wrong is not the reading. It is the SOURCE: every pack opens with a Basic Lease Information summary sheet that restates eight of the eighteen terms and says on its own face that it is not part of the Lease, a rent schedule at the back that repeats what an Article grants, and — on 12 of these 50 packs — an amendment that is a different row's job entirely. A term read off any of those can be perfectly right and cite a page that binds nobody.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 0 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one page is

cut on the === PAGE n === markers the generator writes; a pack with none falls back to one whole-document page

SetupWhat the setup figure measured

There is no index. Preparation is page segmentation only — 50 packs cut into 416 pages by one regex, in process, with no model and no network. Nothing is retrieved: pages exist here so that a drafted value can be checked against the page the abstract SAYS it came from.

LicenceLicence

MIT — this repository's own licence. Every tenant name, building name, file reference, figure and clause is invented; no executed lease, published lease form, industry standard form, real landlord, tenant, property, broker or lease administrator is named or reproduced. Every pack states on its FIRST LINE that it is synthetic.

Bring your ownBring your own lease packs

Replace data/corpus/*.txt with packs carrying page markers, write your own data/fields.json, and supply a gold row per pack WITH A PAGE PER FIELD. src/rulebook.py is the FIRST thing to replace: governs() keys on two page headings this corpus invented, and the precedence between an Article and an Exhibit is this kit's own simplification. Because six things import that one module — the corpus generator, the prompt, the extractor, the scorer, the pre-flight and both free floors — replacing it moves all of them together.

⚠︎ And what stops being true when you do: Three of this kit's five graders need labels and go away on your own leases: the value grade, the citation grade and the absence matrix. What survives is the two NO-GOLD checks — does the cited page state the value, and does that page govern — and this kit measures exactly how much the first one misses: the free summary-sheet floor's citations verify 400 times out of 400 and every one of them is wrong. ⚠︎ AND THE SECOND FREE FLOOR IS A WARNING ABOUT THIS CORPUS RATHER THAN A COMPLIMENT TO REGEXES. It reaches 97.67 pct value accuracy because its patterns are tuned to this corpus's exact sentence shapes, which a generated corpus has and a real pack does not. Read its citation figure (83.53 pct), which is about a HEURISTIC and transfers; do not read its value figure, which is about a template and does not.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 3,565 tokens assembled, 2,110 are contexts — 59% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured token-for-token via evals/prompt_tokens.py's nested-prefix method on LSE-0027 — three calls at max_tokens=1, each part's size the difference between two consecutive prompt_tokens counts the provider itself returned. The three parts sum to 3565, which is exactly the input_tokens the captured example call on the same pack was billed. kits/UC0081-lease-abstract/results/tokens-p001-lease-abstract.json.

Full promptThe prompt in full

You are a lease administrator drafting the abstract of one commercial lease document pack. You return JSON and nothing else.

EVERY FIELD IS A PAIR. For each field return an object with exactly two keys:
  "value" -- the term itself, or null if the Lease does not state it;
  "page"  -- the number of the page of THIS pack the term is stated on, or null when the value is null.
The pack is supplied page by page, each page introduced by a line reading `=== PAGE n ===`. `page` is that n, as a bare integer.

RULES, in order of importance:
1. A VALUE WITH NO PAGE IS NOT AN ABSTRACT ENTRY. The page is what lets the administrator of record confirm the entry without re-reading the pack, and it is the point of this task. Never return a value with a null page, and never return a page beside a null value.
2. THE `BASIC LEASE INFORMATION` PAGE DOES NOT GOVERN, EVEN WHERE IT IS RIGHT. It is a summary prepared for the convenience of the parties and says so on its face. Where it and an Article disagree, THE ARTICLE'S VALUE IS THE ANSWER. Where they agree, THE ARTICLE'S PAGE IS STILL THE CITATION -- you are recording where the Lease says it, not where it was convenient to read it.
3. AN ARTICLE GOVERNS OVER AN EXHIBIT THAT REPEATS IT. A rent schedule that restates the first year's rent is a schedule; the Article that grants the rent is the citation.
4. AN EXHIBIT GOVERNS WHERE THE ARTICLE DEFERS TO IT. Where an Article states no figure at all and refers you to an Exhibit, the Exhibit page is the citation. Citing the Article there is citing a cross-reference rather than a term.
5. THIS ABSTRACT IS OF the Lease as originally executed. If the pack contains a page headed `FIRST AMENDMENT TO LEASE`, DO NOT apply it and DO NOT cite it. Its recitals restate terms that have not changed and its operative clauses change terms that have; neither belongs in this abstract. The Lease as originally executed is what you are abstracting.
6. AN ABSENCE IS AN ANSWER, AND A MARKET-STANDARD FIGURE IS NOT. If the Lease grants no renewal option, `renewal_notice_days` and `renewal_term_months` are null with null pages -- not 180, not 270, not whatever a lease of this kind usually says. The same is true of an operating-expense base year on a lease that has none.
7. COPY VALUES VERBATIM where the field is a quotation of the Lease. Report every money, area, percentage, day and month figure as a bare number with no currency symbol, no thousands separator and no unit. Report every date as YYYY-MM-DD.
8. Use the exact allowed value for a field that lists them.
9. Return every field named in the schema, even when the answer is null.

Draft the abstract. Return one object per field, each with a "value" and a "page":
- tenant_legal_name (string) -- the tenant's full legal name exactly as the operative article writes it, including the entity suffix
- premises_suite (string) -- the suite or unit designation of the demised premises, verbatim -- e.g. 'Suite 410'
- rentable_area_sf (number) -- the rentable area of the premises in square feet, as a bare number with no separators and no unit
- commencement_date (date) -- the Commencement Date of the Term, as YYYY-MM-DD
- expiration_date (date) -- the Expiration Date of the Term, as YYYY-MM-DD
- term_months (number) -- the length of the initial Term in whole months, as a bare number
- base_rent_monthly_usd (number) -- the monthly instalment of Base Rent payable in the FIRST year of the Term, in dollars to the cent, as a bare number with no currency and no separators
- rent_escalation_pct (number) -- the fixed annual percentage increase in Base Rent, as a bare number without the per-cent sign. Where the rent article states no figure and refers the reader to an Exhibit, the figure and the citation both come from that Exhibit
- security_deposit_usd (number) -- the cash Security Deposit, in dollars to the cent, as a bare number. This is NOT the face amount of any letter of credit the lease also mentions
- permitted_use (string) -- the permitted use of the premises, copied verbatim from the use article and no longer than the sentence that states it
- opex_method (enum) one of: base_year, triple_net, gross -- how the lease charges operating expenses: base_year when the tenant pays increases over a stated base year, triple_net when the tenant pays its share of all operating expenses, gross when operating expenses are the landlord's and the tenant pays none
- opex_base_year (number) -- the calendar year the operating-expense base is measured from, as a bare four-digit number. Stated only on a base_year lease; null on a triple_net or gross one -- return null rather than the commencement year or a guess
- assignment_consent_standard (enum) one of: not_to_be_unreasonably_withheld, sole_discretion, prohibited -- the standard the landlord's consent to an assignment is held to: not_to_be_unreasonably_withheld, sole_discretion when consent may be withheld in the landlord's sole and absolute discretion, prohibited when assignment is not permitted at all
- holdover_pct (number) -- the percentage of Base Rent payable during a holdover, as a bare number without the per-cent sign
- renewal_notice_days (number) -- how many days before expiration the tenant must give notice to exercise a renewal option, as a bare number. Null when the lease grants no renewal option -- return null rather than a market-standard figure
- renewal_term_months (number) -- the length in whole months of the renewal term the option grants, as a bare number. Null when the lease grants no renewal option
- termination_option (enum) one of: yes, no -- does the lease grant the tenant an early termination option? Answer strictly from the options article
- administrator_of_record (string) -- the lease administrator of record named on the execution and routing page -- the staff identifier this abstract is routed to for confirmation, copied verbatim

Return a JSON object with exactly these keys: tenant_legal_name, premises_suite, rentable_area_sf, commencement_date, expiration_date, term_months, base_rent_monthly_usd, rent_escalation_pct, security_deposit_usd, permitted_use, opex_method, opex_base_year, assignment_consent_standard, holdover_pct, renewal_notice_days, renewal_term_months, termination_option, administrator_of_record
Each key's object has exactly the keys "value" and "page". Use null for both when the Lease does not state the term.

LEASE DOCUMENT PACK
-------------------
=== PAGE 1 ===
BASIC LEASE INFORMATION
-----------------------

This summary is prepared for the convenience of the parties and their brokers. It is not
part of the Lease, it does not amend or qualify any provision of the Lease, and in the
event of any inconsistency the provisions of the Lease govern.

    Tenant:                Pinnery Bioworks LLC
    Building:              Larkstone Plaza
    Premises:              Suite 1710
    Rentable Area:         4,100 rentable square feet
    Commencement Date:     October 26, 2020
    Expiration Date:       October 25, 2025
    Term:                  60 months
    Monthly Base Rent:     $20,500.00 per month, first Lease Year
    Security Deposit:      $20,500.00

Prepared by the leasing agent. Not to be relied upon as a statement of the terms of the
Lease.
=== PAGE 2 ===
RECITALS
--------

THIS LEASE AGREEMENT is made between the Landlord of Larkstone Plaza and the Tenant
named in Article 1 below. Landlord is the owner of the Building. Tenant desires to lease
the Premises from Landlord on the terms set out in this Lease. The parties therefore
agree as follows.

Capitalised terms used in this Lease have the meanings given to them in the Article in
which they first appear. The Basic Lease Information sheet bound at the front of this
Lease is a summary only.

Landlord and Tenant have negotiated the terms of this Lease at arm's length, each with
the benefit of its own advisers, and neither party is to be treated as the draftsman of
it for the purposes of any rule of construction.

The Building is held by Landlord subject to the matters recorded against the title to
it, and Tenant takes the Premises subject to those matters so far as they affect the
Premises.

Nothing in this Lease creates a partnership, joint venture or agency between the
parties, and no provision of it is intended to confer a benefit on any person who is not
a party to it.

The headings and the article numbering used in this Lease are for convenience of
reference only and do not affect the construction of any provision.
=== PAGE 3 ===
ARTICLE 1. PREMISES
-------------------

1.1 Demise. Landlord leases to Pinnery Bioworks LLC ("Tenant"), and Tenant leases from
Landlord, the premises designated Suite 328 in the Building, consisting of 4,100
rentable square feet as measured in accordance with Section 1.3.

1.2 Common Areas. Tenant shall have the non-exclusive right to use the common areas of
the Building in common with Landlord and other tenants, subject to the rules and
regulations Landlord adopts from time to time.

1.3 Measurement. The rentable area stated in Section 1.1 has been determined by
Landlord's architect and is conclusive for all purposes of this Lease, and is not
subject to remeasurement during the Term.

ARTICLE 2. TERM
---------------

2.1 Term. The initial term of this Lease (the "Term") is 60 months, commencing on
October 26, 2020 (the "Commencement Date") and expiring at 11:59 p.m. on October 25,
2025 (the "Expiration Date"), unless sooner terminated in accordance with this Lease.

2.2 Delivery. Landlord shall deliver the Premises to Tenant on or before the
Commencement Date in the condition required by Exhibit A. A delay in delivery does not
extend the Term or alter the Expiration Date.
=== PAGE 4 ===
ARTICLE 3. RENT
---------------

3.1 Base Rent. Tenant shall pay Base Rent for the Premises in advance on the first day
of each calendar month. The monthly instalment of Base Rent for the first Lease Year is
$20,500.00, calculated at $60.00 per rentable square foot per annum.

3.2 Escalation. Base Rent shall increase on each anniversary of the Commencement Date by
the fixed percentage set out in the rent schedule attached as Exhibit B. No figure is
stated in this Article.

3.3 Late Charge. Any instalment of Base Rent not paid within five days of its due date
bears a late charge of five percent of the overdue amount.

ARTICLE 7. ASSIGNMENT AND SUBLETTING
------------------------------------

7.1 Assignment. Tenant shall have no right to assign this Lease or to sublet the
Premises in whole or in part, and any purported assignment shall be void.

7.2 Recapture. Landlord may, within thirty days of receiving a request for consent,
elect to recapture the portion of the Premises proposed to be assigned or sublet, in
which event this Lease terminates as to that portion.
=== PAGE 5 ===
ARTICLE 6. OPERATING EXPENSES
-----------------------------

6.1 Operating Expenses. This Lease is a full service gross lease. Operating Expenses for
the Premises are the obligation of Landlord and Tenant shall pay no share of them, and
no base year applies.

6.2 Statements. Landlord shall deliver to Tenant a statement of Operating Expenses for
each calendar year within one hundred and twenty days after the end of that year.

ARTICLE 5. USE
--------------

5.1 Permitted Use. Tenant shall use the Premises for general office use together with an
ancillary training facility.

5.2 Compliance. Tenant shall comply with all laws applicable to Tenant's particular use
of the Premises and shall not cause or permit any nuisance or waste in or about the
Building.

ARTICLE 8. HOLDING OVER
-----------------------

8.1 Holdover. If Tenant remains in possession of the Premises after the Expiration Date
without Landlord's written consent, Tenant shall pay for each month of holding over an
amount equal to 125 percent of the Base Rent payable in the last month of the Term, and
Tenant's occupancy shall be a tenancy at sufferance.
=== PAGE 6 ===
ARTICLE 4. SECURITY DEPOSIT
---------------------------

4.1 Deposit. On execution of this Lease Tenant shall deposit with Landlord the sum of
$20,500.00 in immediately available funds as security for Tenant's performance (the
"Security Deposit"). The Security Deposit is not an advance payment of Rent and is not a
measure of Landlord's damages.

4.2 Application. Landlord may apply the Security Deposit to cure any default by Tenant,
and Tenant shall restore it to its full amount within ten days of written demand.

ARTICLE 9. OPTIONS
------------------

9.1 Renewal. Provided Tenant is not then in default, Tenant may extend the Term for one
further period of 36 months by giving Landlord written notice not less than 365 days
before the Expiration Date. Base Rent for the renewal term shall be the then fair market
rent for comparable space in the Building.

9.2 Early Termination. Tenant has no right to terminate this Lease before the Expiration
Date except as expressly provided in Article 12 (casualty) and Article 13
(condemnation).

9.3 Letter of Credit. Any termination fee or restoration obligation may be secured by a
clean irrevocable letter of credit in the face amount of $20,500.00, which is separate
from and additional to the Security Deposit and is not applied against it.
=== PAGE 7 ===
EXHIBIT A. DESCRIPTION OF THE PREMISES AND DELIVERY CONDITION
-------------------------------------------------------------

The Premises comprise the whole of Suite 328 on the floor shown on the plan bound with
this Exhibit, together with the corridors and service risers serving it exclusively.
Landlord shall deliver the Premises broom clean, with building standard lighting
operational, the base building mechanical systems in good working order, and all common
area path-of-travel work required by applicable law completed at Landlord's cost.

Landlord makes no representation as to the suitability of the Premises for Tenant's
particular use, and Tenant accepts the Premises in their condition as delivered save for
latent defects notified within twelve months.

EXHIBIT B. RENT SCHEDULE
------------------------

Base Rent is payable monthly in advance. The monthly instalment for the first Lease Year
is $20,500.00. On each anniversary of the Commencement Date the monthly instalment then
payable increases by 4.00 percent, compounded annually for the remainder of the Term.

No rent abatement, free rent period or tenant improvement allowance applies except as
separately agreed in writing.
=== PAGE 8 ===
EXECUTION AND ROUTING
---------------------

IN WITNESS WHEREOF the parties have executed this Lease as of the date first written
above. Signed for and on behalf of Landlord by its authorised signatory, and signed for
and on behalf of Tenant by its authorised signatory.

    Lease file reference:          LSE-0027
    Administrator of record:       LA-8872, Portfolio Team Quicksilver

This pack is routed to the administrator of record named above. No term of this Lease is
committed to the lease register until that administrator has confirmed it against the
page of this pack on which it appears.

Raw responseThe raw response

{
  "tenant_legal_name": {"value": "Pinnery Bioworks LLC", "page": 3},
  "premises_suite": {"value": "Suite 328", "page": 3},
  "rentable_area_sf": {"value": 4100, "page": 3},
  "commencement_date": {"value": "2020-10-26", "page": 3},
  "expiration_date": {"value": "2025-10-25", "page": 3},
  "term_months": {"value": 60, "page": 3},
  "base_rent_monthly_usd": {"value": 20500.00, "page": 4},
  "rent_escalation_pct": {"value": 4.00, "page": 7},
  "security_deposit_usd": {"value": 20500.00, "page": 6},
  "permitted_use": {"value": "Tenant shall use the Premises for general office use together with an ancillary training facility.", "page": 5},
  "opex_method": {"value": "gross", "page": 5},
  "opex_base_year": {"value": null, "page": null},
  "assignment_consent_standard": {"value": "prohibited", "page": 4},
  "holdover_pct": {"value": 125, "page": 5},
  "renewal_notice_days": {"value": 365, "page": 6},
  "renewal_term_months": {"value": 36, "page": 6},
  "termination_option": {"value": "no", "page": 6},
  "administrator_of_record": {"value": "LA-8872, Portfolio Team Quicksilver", "page": 8}
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Turn a lease pack into a checkable abstract — 50 lease packs. One model answered, and every answer was then graded Five different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Run it twiceThe same set, run again

THIS IS NOT A REPEATABILITY MEASUREMENT AND IT IS NAMED AS ONE ONLY BECAUSE THE FIELD EXISTS. The two runs differ by the four source-precedence rules in the system prompt, not by chance, so the difference between them measures the PROMPT and says nothing about variance. THE ABLATION MOVED SEVEN CITATIONS AND COST TWO DOCUMENTS. Removing the four source-precedence rules took the citation grade from 100.00 pct to 99.13 pct — 838 of 838 against 799 of 806, on a smaller denominator because 2 pack(s) were lost to the output ceiling under the vaguer prompt. Read it per SHAPE rather than as a headline, because the four rules are not one rule: the misses are amendment_changes_the_term (6 of 16 cited wrong); stale_cover_restatement (1 of 12 cited wrong), and everything else held exactly — absence (58 of 58); amendment_recites_the_term (33 of 33); cover_agrees (323 of 323); exhibit_defer (9 of 9); single_statement (413 of 413). So the SCOPE rule is doing real work and the cover-sheet rule bought nothing this corpus can detect: told only to return a page and never told that the summary sheet does not govern, the model cited the Article on every restated cell anyway. Fifty packs cannot separate ‘the model already holds that convention’ from ‘these traps are easy for it’, and this kit does not claim to have. Whether the shipped prompt would score 100.00 pct a second time is UNMEASURED: the fifty packs were fired once on it, and one clean board is not evidence of a rate.

page-citation accuracy — measured ONCE on the shipped prompt and ONCE on an ablation of it, never twice on the same prompt — Averaging two prompt VARIANTS would publish a number nobody measured for a prompt that does not exist. The two runs are an experiment, not a sample.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran these two runs publishes no rate card this repo commits, so nothing here is what was actually paid — the real spend for this kit's build is recorded in the commit history and the shared call ledger, not on this page.

The gradersFive ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same pack, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The two scored runs' own token counts (50 packs each) priced at Google Gemini 3 Flash's published rate — the same basis cost_per_query_usd uses, not a second, larger spend. Both free floors, the stub, the scorer and the rescorer are pure code and cost $0.00 against any result set. A forker re-running the eval pays this on top of the pipeline.

Cost driversWhat actually moves the bill

• THE DOCUMENT, and this is the driver that inverts this kit against its siblings: the lease pack is 2110 of 3565 input tokens on the measured pack (59 pct) because src/select.py sends every page on purpose. A sibling that maps fields to sections sends about a seventh of its document and pays for its instructions instead.
• OUTPUT, which is 3369 tokens a pack on average against 3506 input — and 87 pct of the captured reply's output was reasoning the provider reported and nobody reads. The model is not copying values, it is ruling pages out, and that is what the reasoning is.
• Nothing amortises. Each pack is an independent call carrying the whole fixed prefix again, with no cache and no batch.

Your volumeWhat it costs at your volume

Linear in packs: each call is independent, carries the same fixed prompt and shares nothing with its neighbours, so 500 packs cost ten times 50 and take ten times as long. Nothing amortises — there is no index to build and no cache. The wall-clock number bites first: 50 packs took 1251.3 seconds serially, so a portfolio acquisition of a few hundred leases is hours before anything is parallelised. AND THE PER-PACK COST IS NOT CONSTANT AT REAL LENGTHS: this corpus is 8 to 10 pages and a real pack is sixty to two hundred, so the input alone is an order of magnitude larger before any of this arithmetic applies.

Model choiceWhy this model

ONE tier, and the second run spent on an ABLATION rather than on a second model. The citation grade came back at 100.00 pct on the first run, and a score that high makes ‘would a bigger model score higher’ the least informative question available — there is no headroom for it to answer in. The question worth fifty calls was whether the corpus and the prompt were measuring anything at all, which only an ablation can answer. What this kit therefore does NOT have is a tier comparison, and that is named in could_not_verify rather than implied by silence.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 50 lease packs, one completion call each. The ablation's own 48 calls are the same tier on a shorter prompt and are recorded separately in Cost.cost_by_model; they are not projected here.

The codeEvery file, and what it is for

Seven modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/pages.pypages — a swap seam

cut the pack into addressable PAGES, pure code — and answer, with no labels at all, whether a cited page really states a value, across every spelling the same value can take (a lease writes January 1, 2021 and an abstract records 2021-01-01)

You change it to: how a pack is cut into pages. This corpus writes === PAGE n ===; a real pack has printed page numbers or PDF page boxes. With no marker the fallback is ONE page, at which point every citation is page 1 and the citation grade means nothing

# Cut a lease document pack into PAGES. Pure code — no model, no network.

def pages(text):
def page_by_number(pgs, n):
def _date_variants(s):
def variants(value):
def locate(text, value):
def cited_page_confirms(pgs, page, value):
def where_stated(pgs, value):

src/select.pyselect — a swap seam

decide what reaches the model, which in this kit is EVERY PAGE on purpose — pre-selecting the page a term lives on and then grading the page the model names would measure the map in this file rather than the model

You change it to: the no-op. It sends every page deliberately; a longer pack may force you to select, and the moment you do, the citation grade starts measuring your selection as much as the model

# Decide what reaches the model. In this kit that is EVERY PAGE, and the no-op is the point.

ARTICLE_OF = {
def for_pack(pgs):
def plan(pgs, fields):

src/rulebook.pyrulebook — a swap seam

which page governs, and the per-field interlock, as the ONLY copy in the kit. Six things import it, so there is no second implementation to drift

You change it to: which pages govern your packs, and what your interlock requires before an abstract may be routed. governs() keys on two page headings this corpus invented; the Article-over-Exhibit precedence and the as-originally-executed scope are this kit's own simplifications. Six things import this one module — the corpus generator, the prompt, the extractor, the scorer, the pre-flight and both free floors — so changing it here changes all of them together. THIS IS THE FIRST THING A FORKER REPLACES.

# THE ABSTRACTION RULE, and the only copy of it in this kit.

COVER_HEADING = "BASIC LEASE INFORMATION"
AMENDMENT_HEADING = "FIRST AMENDMENT TO LEASE"
ABSTRACT_SCOPE = "the Lease as originally executed"
CONFIRMATION_REQUIRED = True
CONFIRMED = "confirmed_by_page"
FULL_READ = "needs_full_read"
BLOCKED = "blocked"
def governs(page_heading):
def page_in_range(page, n_pages):

src/prompt.pyprompt

assemble one call for all 18 fields, each asked for as a {value, page} pair, with the four source-precedence rules stated in full — and the ablated variant that removes exactly those four and nothing else

# Assemble the abstraction prompt. One prompt per lease pack, all eighteen fields in it.

PRECEDENCE_RULES = (2, 3, 4, 5)
def _system(ablate_precedence=False):
def _assemble(ablate_precedence):
SYSTEM = _system()
SYSTEM_ABLATED = _system(ablate_precedence=True)
def field_schema(fields):
def build(doc_text, pgs, fields, selector, ablate_precedence=False):
def _page_int(v):
def parse(raw, fields):

src/extract.pyextract

the AI layer, one provider one key — plus the pure-code interlock downstream, and the second no-gold check (does the cited page govern) that is deliberately NOT part of the shipped interlock

# Abstract one lease document pack: paginate, prompt with the whole pack, one model call, then a

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 12000
def load_fields():
def load_doc(pack_ref):
def documents():
def interlock(cells, pgs, fields):
def cited_page_governs(pgs, page):

evals/judge.pyjudge

score value, citation, the no-gold verification, the absence matrix and commit-readiness separately, pure code, plus a per-shape breakdown

# Score a lease-abstract run. PURE CODE -- gold is exact, every answer is a value and a page, and

def norm(v):
def _num(v):
def equal(field, got, want):
def boundary_tolerant(cells):
def _shape_of(g, field):
def score(fields, records, golds, pages_by_doc):
def _matrix(rows, positive):
def score_abstracts(fields, records, golds, pages_by_doc, locks):

src/app.pyapp

the local UI on 127.0.0.1:9006, standard library only — it renders the whole abstract with no API key at all

# The minimal local UI. Standard library only -- python -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
PORT = int(os.environ.get("PORT", "9006"))
class H(BaseHTTPRequestHandler):
def main():

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 3505 input and 3369 output tokens per pack, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's lease packs are entirely synthetic (tools/build_corpus.py, seed 20260823): no real lease, landlord, tenant, property, broker or administrator exists in the corpus, and nothing was fetched from anywhere. The only outbound traffic the kit makes is one chat-completion request per pack, carrying every page of that pack. Nothing is written outside the kit directory, there is no database, no auth and no multi-tenancy, and the local UI binds 127.0.0.1 only.

Read from the shared .env, this kit's own .env (which holds one MODEL line and no credential) or the real environment only, never committed, never logged and never rendered. .env is gitignored from the first commit. On an adapter error the app redacts the key and base URL out of the message before returning it, so a misconfigured endpoint cannot echo a credential into a browser.

The experimentWe did not attack it -- and four of five boundaries hold

The four boundaries that hold were confirmed by reading the code, not by a run: no code path commits anything or writes at all; governs() branches on a page's own heading and never on its content, so neither the summary sheet nor an amendment can reach the citation rule; the corpus generator forces both of those pages onto pages of their own so a heading is never another block's; and the key and base URL are redacted out of any error before it is returned. The fifth is open. An indirect prompt injection needs a page an outside party authored, and a real lease pack has three classes of them. Confirmed by reading the code, not by a run, on 2026-08-23 -- no attack run was fired; see redteam.why.

The first four boundaries hold, confirmed by reading the code rather than by an attack run. The fifth is open and is named as open — an indirect prompt injection inside a lease clause is exactly the attack this kit's shape invites, and the free summary floor has already demonstrated, with no attacker at all, that a wrong PAGE passes every no-gold check this kit ships. An attacker would not need to move a single value.

The result0 attack trials, and four of five boundaries checked here hold on evidence. The one that does not is the one this kit's own shape invites — and this kit has already shown, without any attacker, what an attack on it would look like: the free summary-sheet floor's citations verify 400 times out of 400, the interlock passes every one of its abstracts, and every citation is wrong.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260823), so no text in it came from an outside party and there was nothing adversarial to resist. The planted shapes are a RESTATEMENT that agrees, a restatement that is STALE, and a SCOPE boundary — all three measure whether a model cites the page that grants a term when other pages state it. None measures whether a model obeys an instruction hidden in one of those pages. Those are different failures and only one kind is measured here.

HonestyWhat this does not prove

• Whether an indirect prompt injection in a recital, a permitted-use clause or an exhibit could move a page citation on this tier. No attack run exists — and this is the kit where that gap matters most, because moving the page is enough to defeat every no-gold check here without moving a value.
• Whether the kit behaves safely against a hostile provider — a response body crafted to break the JSON extraction in src/prompt.py::parse, or to return an enormous payload. parse() fails closed to an empty dict, which the harness records as a failed pack, and nothing beyond that was tested.
• Whether a real lease archive would carry anything sensitive this kit mishandles. The corpus has no personal or commercial data by construction — the administrator of record is a staff identifier rather than a person's name, deliberately — and a real pack carries named parties, a negotiated rent and an executed signature block. None of that path is exercised.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

Every drafted field is routed to the administrator of record named on the pack's own execution page, and the abstract is commit-READY only when NO field is blocked. A field is `confirmed_by_page` when it has a value, a page in range, and the pack confirms the value is on that page; `needs_full_read` when the value is null and no page is cited, because confirming an ABSENCE means reading the pack rather than one page; and `blocked` otherwise — a value with no page, a page outside the pack, a page that does not state the value, or a page cited beside a blank. The rule runs in pure code over whatever the model returned, and never over gold.

src/rulebook.py::field_state and ::commit_ready, called by src/extract.py::interlock on every pack, by the local app on every click, and by evals/baseline.py on both free floors' own output, so all four are judged by identical code.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• ⚠︎ IT IS NOT A CHECK THAT THE CITED PAGE GOVERNS, AND THAT IS MEASURED RATHER THAN ADMITTED. It asks whether the page CONTAINS the value; it cannot ask whether the page GRANTS the term without already knowing which article does, which is the question. The free summary-sheet floor is the demonstration: 400 of 400 of its citations verify, every one of them names a page that binds nobody, and the interlock passes all 50 of its abstracts.
• ⚠︎ IT IS NOT A CHECK THAT THE ABSTRACT IS COMPLETE. A null is the assertion of an absence, so an abstract that drafts two fields of 18 and leaves 16 blank passes it. Measured on the free stub run, which is commit-ready on all 50 packs. The absence matrix is the instrument for that and it scores the stub at 0.08 precision.
• It is NOT a check that the VALUES are right. It passed 50 of 50 abstracts on the shipped run and 11 were correct end to end.
• It is NOT a real lease-administration policy. ‘Every field carries a page citation the pack confirms’ is this kit's own simplification, invented for this corpus. No lease-administration standard, portfolio procedure or abstract template was consulted, and none is reproduced.
• It is NOT a disposition. commit_ready means the record is complete enough to put in front of the administrator of record. It is never a commit, and the two words are the whole guardrail.

WatchedWhat is watched, and why that one

The kit measures 30 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

10 measured by the latest run 20 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (absence_precision 1.0, absence_recall 1.0, max_tokens 12000, provider openai-compatible, thinking True) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

4 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked.

not a time series No two of these 4 runs measured the same system — they differ on absence_precision, max_tokens, packs, provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

DeviationsWhat deviated

0 breaches across 4 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the interlock's evidence on any change to src/rulebook.py, to data/fields.json's literal flags, or on any corpus regeneration — a repagination moves gold's pages, which are the labels the citation grade is measured against. Changing what the interlock REQUIRES is a policy change and must be re-scored even though it never changes which page governs. And re-check on any provider or model change with more weight than a sibling kit would: the citation grade here has no headroom, so a change can only show as a regression.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt is empty on purpose: the corpus is generated in-process, the pages are cut with one regex, the provider is reached over urllib, and the UI is one HTML file with no build step. A forker runs this on whichever key they already hold, with one clone and no install.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph. The pipeline has one path per pack — paginate, prompt, one call, parse, interlock — with no branch, no loop and no agent. What looks like depth in this kit is inside the model's reasoning, not in the orchestration: 87 pct of the captured reply's billed output was the model ruling pages out, and none of that is code anybody wrote.

The other sideWhat a framework costs you

• Everything is hand-rolled, so everything is yours to maintain: the JSON extraction from a fenced reply, the retry/backoff policy, the page regex, the numeric and date variant search and the .env reader are all code somebody has to own.
• No framework means no framework's ecosystem — no tracing, no eval harness beyond the one in evals/, no prompt registry, no schema validation library. What ships is what is in the repository.
• The provider abstraction covers exactly two wire formats. A third provider is one function and one dict entry.
• Sending every page rather than selecting is a deliberate choice and it costs what this page measures: the pack is 59 pct of the input. A kit that selected would be cheaper and would measure its own map instead of the model, which is the question this kit exists to ask.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-lease-abstract on the same tier, source-precedence rules REMOVED, 2026-08-23. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 3 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

1 run not plotted. b000-cover recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — each document goes whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT — this repository's own licence. Every tenant name, building name, file reference, figure and clause is invented; no executed lease, published lease form, industry standard form, real landlord, tenant, property, broker or lease administrator is named or reproduced. Every pack states on its FIRST LINE that it is synthetic. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe page each field cites, against the page that governs

Did the abstract cite the page the term is actually granted on? THIS IS THE HEADLINE AND IT IS A DIFFERENT NUMBER FROM VALUE ACCURACY BY CONSTRUCTION: 465 of the 838 citable cells state their value on some OTHER page as well, 422 of them on a summary sheet that binds nobody. A right value read off the wrong page scores 100 pct on the value grade and 0 pct here.

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model. The same function every free floor and the ablation are scored by — a baseline and a paid run scored by two different scorers cannot be compared honestly.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, whose PAGE is LOCATED rather than typed: the generator paginates the pack and then finds the governing sentence in the finished text. evals/check_labels.py re-derives every one from the packs on disk and refuses the run if a single citation points at a page that does not state its value or at a page src/rulebook.py::governs rules out.

How tight can the band be? No threshold — the cited page equals the governing page or it does not. A page cited for a null value is over_cited and a value with no page is uncited; neither is folded into the correct cell.

Cadence: Re-score on any change to src/rulebook.py, src/prompt.py, data/fields.json or tools/build_corpus.py. Re-run evals/run.py (paid) on any change to the prompt variant, MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one linePer-field exact match against gold

Does the abstract's value for each field match gold, after trimming trailing punctuation and collapsing whitespace, with numbers to half a hundredth? What every extraction kit publishes — and on this kit it is the SECOND figure, not the first.

MethodHow the test was run

evals/judge.py::score and ::boundary_tolerant, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, read back off the same values the pack's operative article states; evals/check_labels.py asserts every non-nullable field is populated on every row and that all three nullability invariants hold as iff-conditions before any run is allowed to spend.

How tight can the band be? Exact after trimming trailing punctuation and collapsing whitespace; numbers to half a hundredth. ⚠︎ A SECOND, BOUNDARY-TOLERANT FIGURE IS REPORTED BESIDE IT AND NEVER IN PLACE OF IT (evals/judge.py::boundary_tolerant). It was added AFTER reading the shipped run, which is stated rather than hidden: relaxing a grader after seeing what it convicted is the oldest way to publish a better number than you measured.

Cadence: Re-score on any corpus regeneration. Re-run (paid) on any change to the prompt or the field hints — and note that fixing the span-boundary gap IS a prompt change and invalidates every figure on this page.

The decisionWhen to reach for it

In one lineDoes the cited page actually state the value

PURE CODE AND NO LABELS: does the page this entry cites contain this value, across every spelling the same value can take? It is the ONE figure a forker keeps on their own leases — and this kit exists to measure how much it misses.

MethodHow the test was run

evals/judge.py, off src/pages.py::cited_page_confirms. In-process, no key, no model, and NO GOLD — that is the property being demonstrated.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The pack on disk. src/pages.py::locate searches the cited page for the value as written, at two decimals, as a bare integer, with thousand separators, and — for a date — in the forms a lease writes it in. A run of spaces matches any whitespace, because a lease is WRAPPED text and the first build of this corpus convicted its own gold on all fifty packs for exactly that.

How tight can the band be? No threshold and no fuzz. Either one of the exact spellings is on the page or the citation does not verify; a tolerant matcher would quietly convert the guarantee into an impression.

Cadence: Re-run on any change to src/pages.py's variant list or on any corpus whose values are written in a different notation.

The decisionWhen to reach for it

In one lineTerms the Lease does not grant, against terms the abstract left blank

Of every term the Lease genuinely does not grant, how many did the abstract correctly leave blank — and of every blank it drew, how many were real? AN INVENTED TERM IS THE POSITIVE FAILURE: a market-standard renewal notice period written onto a portfolio record for a lease with no renewal option is the abstraction error that costs money years later.

MethodHow the test was run

evals/judge.py::score_abstracts, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: Gold's nulls, each governed by an iff-invariant evals/check_labels.py asserts before any run may spend: a renewal notice period exists exactly when a renewal term does, and an operating-expense base year exactly on a base_year lease.

How tight can the band be? No threshold — a cell is blank or it is not. A cell the reply omitted entirely is counted unanswered rather than folded into the correct-negative cell.

Cadence: Re-run on any corpus regeneration, since the null count is a property of the drawn shapes.

The decisionWhen to reach for it

In one lineAbstracts the interlock passed, against abstracts that were right

Of 50 abstracts, how many are correct END TO END — every value matching gold AND every citation on the governing page — and HOW MANY DID THE PURE-CODE INTERLOCK PASS THAT WERE NOT? That second number is false_confidence and it is the finding this kit exists to produce.

MethodHow the test was run

evals/judge.py::score_abstracts, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/rulebook.py::commit_ready, the same function the run applies, against gold's own values and pages. One rule, so a change to the interlock moves both sides together and this grader cannot silently grade an old policy.

How tight can the band be? No threshold — an abstract is correct end to end or it is not, and one wrong cell of 18 is enough. That is a deliberately brutal bar and it is why the per-cell figures are published beside it.

Cadence: Re-run whenever src/rulebook.py's interlock changes. Changing what the interlock REQUIRES is a policy change and must be re-scored even though it never changes which page governs.

The decisionWhen to reach for it