The business caseThe problem this solves

An agreement of fifty-odd clauses is drafted in pieces, amended in a sheet bolted on the end, and priced in a schedule nobody reads next to the body. The places where it contradicts ITSELF are not in any one paragraph -- they are between two paragraphs eleven pages apart, and both read correctly on their own. Finding them means holding the whole document in your head at once, which is the one thing reading it page by page does not do. Reading one agreement end to end with a pen, twice, holding every figure, definition, deadline and schedule paragraph against every other one.

The inputThe actual agreements, each read whole

The corpus is 60 agreements, each read whole, 0.54 MB (json 2 · jsonl 1 · txt 60). The thing being measured is a NEGATIVE as much as a positive, and no public dataset carries it. What makes this job hard is not spotting that two numbers differ -- that is arithmetic and the free floor does it -- it is knowing when two clauses that disagree on their face are reconciled by words somewhere else in the document. That requires a set where the near misses are LABELLED, and a labelled near miss can only exist in a corpus somebody planted. 24 of these 60 agreements carry no conflict at all and 23 carry a planted near miss, which is what lets the scorer report how many traps each arm walked into by name instead of reporting one accuracy.

SYNTHETIC AGREEMENT -- generated for an AI Foundry evaluation kit. Not a real lease.
No real landlord, tenant, property, managing agent or standard form is named, and none
is reproduced or derived from. MIT licensed. See data/SOURCES.md.
File reference AGR-0001.

LEASE AGREEMENT
===============

This Lease is made between Marlstone Estates LLC (the Landlord) and Ostlerbrook Analytics LLC
(the Tenant) and relates to the premises described in it.

ARTICLE 1. DEFINITIONS
----------------------

1.1   "Account Year" means each period of twelve months ending on 31 March.

1.2   "Business Day" means any day other than a Saturday, a Sunday or a day on which the clearing
      banks are not open for business.

1.3   "Expenditure" means the costs the Landlord properly incurs in the matters listed in Schedule 2
      for an Account Year.

1.4   "Insured Risks" means fire, storm, flood, escape of water, impact and the other risks against
      which the Landlord insures the Building.

1.5   "Lease Year" means each period of twelve months beginning on the Commencement Date or an
      anniversary of it.

1.6   "Premises" means the premises demised by clause 2.1, and does not include the car park, the
      Common Areas or the structure of the Building.

1.7   "Rent" means the Base Rent reserved by clause 4.1, and does not include the Service Charge,
      the Insurance Rent or any other sum payable under this Lease.

1.8   "Term" means the term granted by clause 3.1, and does not include any period during which the
      Tenant holds over after the Expiry Date.

ARTICLE 2. PREMISES
-------------------

2.1   Demise. The Landlord demises to the Tenant the premises known as Suite 1801 in the building
      known as Greyloft Building, comprising 33,900 rentable square feet.

The outcomeWhat a good result looks like

A short list a person CONFIRMS rather than one they build: each finding is two clause numbers and the words in each that clash, so checking it is opening the document at two places. Every citation is re-resolved in pure code first, so a clause number the agreement does not have never reaches the reader.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• A pile of agreements to triage, where a person will read every finding anyway and the cost of missing a contradiction is high — the model
  It found 35 of 39 planted conflicts against the floors' 22, and it is the only arm with any mechanism at all for the conflicts that carry no figure: DEFINED_TERM_DRIFT 3 of 7 against 0 of 7, AMENDMENT_ORPHAN 9 of 9 against 4, IMPOSSIBLE_TOGETHER 6 of 6 correctly typed against 2 found and 0 typed right. If a reader is going to check every line regardless, recall is what you are buying.
• A report somebody signs off on, where a false flag costs a professional's afternoon and a clean document must come back clean — the free carve-out floor, b001
  100.0 pct precision, 0 false positives across all 60 agreements, all 24 clean agreements left clean, and 43 of 60 agreements exactly right -- against the paid arm's 23. It costs $0.00 and runs in 17 s with no network. It misses 39 of the 0 planted conflicts and every one of them has no number in it, so you are choosing which error you would rather have.
• Agreements whose conflicts are figure-against-figure -- a notice period stated twice, a schedule share against a body share — either free floor, and do not pay
  FIGURE_DISAGREEMENT is 10 of 10 on the paid arm and 10 of 10 on both free floors. SCHEDULE_CONTRADICTS_BODY is 7 of 7 against 6 of 7. On the numeric half of this job the call buys one extra pair and costs $1.5695.

And where nothing here is good enough:

• Anything where the answer depends on a second document -- a side letter, a guarantee, an earlier lease, a standard form — neither arm
  The instruction states in terms that there is no second document and the whole kit assumes the answer is inside the file. A cross-document conflict is a different use case with a different unit of work.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-31. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-31 — r001-clause-conflict. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Measured on a copy of the kit with no .env and no API key: python3 -m tools.build_corpus rebuilds all 60 agreements and the 60-row key in 0.08 s, byte-identical to the committed tree; python3 -m evals.check_labels re-grades the key and passes; python3 -m evals.run --run-id b000-clause-conflict-rules --floor rules scores all 60 agreements in 0.2 s. requirements.txt names no package and the code imports none -- standard library end to end.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. one agreement -- one model call, end to end, including provider-side reasoning tokens, on a shared connection. The citation recheck adds nothing measurable: it is pure Python over the reply. ⚠︎ THIS IS NOT AN SLA AND IT IS NOWHERE NEAR INTERACTIVE. p50 347.3 s and p95 590.5 s PER AGREEMENT; the 60 calls summed to 20,730.2 s of call time and finished in 4,238.2 s of wall clock (70.6 minutes) at 5 workers -- the summed-to-wall ratio is 4.89, which is what proves the concurrency rather than a setting anybody read. The two free floors answer all 60 agreements in 0.2 s with no network at all.

Current processWhat it replaces

Reading one agreement end to end with a pen, twice, holding every figure, definition, deadline and schedule paragraph against every other one.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one reports -- one per agreement is

No train/test split, because nothing is trained or tuned, and no segmentation, because the agreement goes whole into one call. The unit every rate stands on is the AGREEMENT: 60 agreements, 60 reports, 39 planted conflicts and 24 planted near misses. Eight case families, fixed by the generator before any arm answered: amendment_orphan 7, clean 14, clean_carveout 10, defined_term 6, impossible_pair 5, multi_conflict 3, notice_two_ways 8, schedule_figure 7. ⚑ THE READING IS IN THE CLEAN/CONFLICTED SPLIT: 24 agreements are clean, and on those the free carve-out floor is 24 of 24 while the paid arm is 4 of 24.

SetupWhat the setup figure measured

There is no index to build and no retrieval step: the whole agreement goes into one prompt at 3,445.5 input tokens on average. The 0.08 s is the corpus generator rebuilding all 60 agreements, the 60-row answer key and the statistics from scratch -- measured this session on a copy of the kit with no .env and no key. ⚑ REPRODUCIBILITY WAS TESTED RATHER THAN ASSERTED: rebuilt under PYTHONHASHSEED 0, 1, 7 and 12345, all four runs are byte-identical to the committed data/corpus/*.txt, data/gold.jsonl, data/agreements.json and data/corpus-stats.json. The kit ships no --check mode, so the test was a rebuild into a scratch copy and a byte diff.

LicenceLicence

MIT -- and the MIT text is in this repository's LICENSE-PUBLIC file, NOT in its root LICENSE, which governs the repository itself and grants a reader nothing. The KIT and its corpus are MIT; the repository is not, and this line says which, because it is the only claim a reader is entitled to rely on before forking. There is no third-party data in it to licence at all: the 60 agreements, the answer key, the near misses and the statistics are generated in-process by tools/build_corpus.py from a fixed seed, and nothing was scraped, licensed, anonymised or derived from anybody's document. Each corpus file repeats the grant in its own header. ⚠︎ The kit ships no data/SOURCES.md to state the same distinction, which is a gap in the kit and not in this claim: LICENSE-PUBLIC was read and checked this session.

Bring your ownBring your own agreements, each read whole

Drop your own agreements into data/corpus/ as .txt with a clause identifier at the start of each paragraph in the printed layout -- 4.2 two spaces then the text for the body, S2.1 for a schedule paragraph, A1.2 for an amendment sheet. The board, the free floors and the model path all read them with no other change. To SCORE them you also need a key: one data/gold.jsonl row per file naming each conflicting pair, its type and a verbatim span from each clause -- and evals/check_labels.py will refuse a key whose span is not in the clause it names.

⚠︎ And what stops being true when you do: ⚠︎ THE WHOLE AGREEMENT REACHES YOUR CONFIGURED PROVIDER VERBATIM, ON EVERY CALL. There is no redaction step, no extraction step and no summary: that is the design, because a summary is where the operative words get lost before the model sees them. If the document is confidential to somebody else -- and an executed commercial agreement usually is -- that is a decision to take before running this, not after. The free floors and the board's non-check routes send nothing anywhere and are the honest way to try it on a real document.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 13,539 characters assembled, 8,685 are documents — 64% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

The exact two parts sent for AGR-0001, the first agreement of run r001-clause-conflict. 13,539 characters total: 4,854 of instruction and 8,685 of agreement.

Full promptThe prompt in full

You are reading ONE agreement and looking for the places where it contradicts itself.

Everything you need is inside the document in front of you. There is no second document, no
standard form, no policy and no other agreement to compare it against: the question is only whether
this agreement's own clauses, schedules and amendment sheets can all be complied with at once.

Return every PAIR of clauses that cannot both be satisfied. A pair is two clause identifiers as the
document itself spells them -- 4.2 for a clause of the body, S2.1 for a paragraph of a schedule,
A1.2 for a paragraph of an amendment sheet.

MOST AGREEMENTS HAVE NO CONFLICT AND MANY CONTAIN A PAIR THAT LOOKS LIKE ONE AND IS NOT. Returning
an empty list is a real answer and often the right one. A report that flags every pair of clauses
whose numbers differ finds all of the conflicts and is worth nothing to the person reading it,
because they now have to check every line of it themselves. Report a pair only where you can say,
in the document's own words, what each clause requires and why both cannot be done.

Each conflicting pair gets exactly one type:

  FIGURE_DISAGREEMENT
      Two clauses in the body of the agreement state different figures, durations or dates
      for the same obligation — a notice period given as six months in the clause that
      grants the right and as nine months in the clause that governs notices, an interest
      rate stated twice, two deadlines for one act.
      NOT THIS: Two figures that govern different things, different periods or different parties are
      not a disagreement, however similar they look. Nor is a pair where one of the clauses
      says, in terms, which figure applies and when.

  SCHEDULE_CONTRADICTS_BODY
      A figure in a schedule or an appendix contradicts the figure the body of the agreement
      states for the same item — a proportionate share, a rent, an area, a count of spaces.
      NOT THIS: A schedule that varies the body for a stated period, or that the body itself defers to
      by name, is not a contradiction: it is the drafting working. Read what the body says
      about the schedule before reporting the pair.

  AMENDMENT_ORPHAN
      An amendment changed a term, and some other clause — one the amendment does not name —
      still relies on the superseded term: it repeats the old figure, or the old date, as
      though the amendment had not happened.
      NOT THIS: An amendment disagreeing with the clause it names is not a conflict. That is what an
      amendment is: the clause it names is superseded, and so is any clause the amendment
      redirects references to. The orphan is the clause left behind, not the one that was
      replaced.

  DEFINED_TERM_DRIFT
      A term the agreement defines is used somewhere with a meaning the definition excludes,
      or is redefined in passing — the definition says a word does not cover something, and
      a later clause says it does.
      NOT THIS: A definition that is narrowed or widened for one clause by express words the agreement
      itself provides for is drafting, not drift. The question is whether both statements
      about the same word can stand.

  IMPOSSIBLE_TOGETHER
      Two clauses impose obligations or permissions that cannot both be satisfied — a
      deadline that expires before the thing it depends on is due, an act one clause
      requires and another forbids, a right granted and removed.
      NOT THIS: A prohibition with an express exception, or an obligation subject to another clause,
      is not impossible. Neither is a pair where one clause is expressed to prevail over the
      other.

For every pair you report, quote the words that actually clash -- the shortest span of each clause
that carries the contradiction, copied exactly from that clause. NEVER write a quote or a clause
identifier that is not in the document in front of you: an invented citation is worse than no
finding at all, because it is the first thing the reader will check.

Reply with JSON and nothing else:

{"conflicts": [{"clause_a": "<identifier from this document>",
                "clause_b": "<a different identifier from this document>",
                "conflict_type": "<one of the five above>",
                "subject": "<what the two clauses are both about, in a few words>",
                "quote_a": "<the words from clause_a that clash, copied exactly>",
                "quote_b": "<the words from clause_b that clash, copied exactly>",
                "why": "<one sentence: why both cannot be satisfied>"}],
 "rationale": "<two sentences at most, on the hardest call you made -- including a pair you
                considered and did NOT report, if there was one>"}

"conflicts" is an empty list when the agreement does not contradict itself. Do not fill it to look
thorough.

THE AGREEMENT, verbatim. Clause identifiers are at the start of each paragraph.
Answer about this document and no other.

SYNTHETIC AGREEMENT -- generated for an AI Foundry evaluation kit. Not a real lease.
No real landlord, tenant, property, managing agent or standard form is named, and none
is reproduced or derived from. MIT licensed. See data/SOURCES.md.
File reference AGR-0001.

LEASE AGREEMENT
===============

This Lease is made between Marlstone Estates LLC (the Landlord) and Ostlerbrook Analytics LLC
(the Tenant) and relates to the premises described in it.

ARTICLE 1. DEFINITIONS
----------------------

1.1   "Account Year" means each period of twelve months ending on 31 March.

1.2   "Business Day" means any day other than a Saturday, a Sunday or a day on which the clearing
      banks are not open for business.

1.3   "Expenditure" means the costs the Landlord properly incurs in the matters listed in Schedule 2
      for an Account Year.

1.4   "Insured Risks" means fire, storm, flood, escape of water, impact and the other risks against
      which the Landlord insures the Building.

1.5   "Lease Year" means each period of twelve months beginning on the Commencement Date or an
      anniversary of it.

1.6   "Premises" means the premises demised by clause 2.1, and does not include the car park, the
      Common Areas or the structure of the Building.

1.7   "Rent" means the Base Rent reserved by clause 4.1, and does not include the Service Charge,
      the Insurance Rent or any other sum payable under this Lease.

1.8   "Term" means the term granted by clause 3.1, and does not include any period during which the
      Tenant holds over after the Expiry Date.

ARTICLE 2. PREMISES
-------------------

2.1   Demise. The Landlord demises to the Tenant the premises known as Suite 1801 in the building
      known as Greyloft Building, comprising 33,900 rentable square feet.

2.2   Common Areas. The Tenant may use the Common Areas in common with the Landlord and the other
      occupiers of the Building, subject to the regulations the Landlord makes from time to time.

2.3   Measurement. The rentable area stated in clause 2.1 was determined by the Landlord's surveyor
      and is conclusive for all purposes of this Lease.

ARTICLE 3. TERM
---------------

3.1   Term. The Term is 72 months beginning on the Commencement Date of March 12, 2022 and ending on
      the Expiry Date of March 11, 2028.

3.2   Yield up. On the Expiry Date the Tenant shall yield up the Premises with vacant possession, in
      the state of repair this Lease requires.

ARTICLE 4. RENT
---------------

4.1   Base Rent. The Tenant shall pay Base Rent of $69,766,200.00 per annum, payable in advance by
      equal monthly instalments on the first day of each month.

4.2   Review. The Base Rent is increased on each Review Date by 4.00 per cent of the Base Rent
      payable immediately before that Review Date.

4.3   Interest. If any sum is not paid within 7 Business Days of the day it falls due, the Tenant
      shall pay interest on it at 3.00 per cent above the base rate from the due date until payment.

ARTICLE 5. UTILITIES
--------------------

5.1   Utilities. The Tenant shall pay for the electricity, gas, water and telecommunications
      supplied to the Premises, and where they are not separately metered shall pay the proportion
      the Landlord properly attributes to the Premises.

ARTICLE 6. SERVICE CHARGE
-------------------------

6.1   Proportionate Share. The Tenant shall pay the Tenant's Proportionate Share of the Expenditure,
      which Proportionate Share is 16.20 per cent.

6.2   Certified statement. The Landlord shall issue a certified statement of the Expenditure for
      each Account Year not later than 90 days after the end of that Account Year.

6.3   Objection. The Tenant may serve notice of objection to a certified statement not later than 45
      days after the date on which that statement is issued.

ARTICLE 7. RENEWAL
------------------

7.1   Option. The Tenant may require the Landlord to grant a new lease of the Premises for a term of
      36 months by notice given not less than 12 months before the Expiry Date.

7.2   Terms of the new lease. The new lease shall be on the same terms as this Lease, except for
      this clause and except that the rent shall be the open market rent at the date of grant.

ARTICLE 8. INSURANCE
--------------------

8.1   Landlord's insurance. The Landlord shall insure the Building against the Insured Risks for its
      full reinstatement cost, and the Tenant shall pay the Insurance Rent.

8.2   Notice of claim. The Tenant shall notify the Landlord of any event which may give rise to a
      claim within 7 Business Days of becoming aware of it.

8.3   Excess. The Tenant shall pay the excess under any policy, which excess is $6,000.00 for each
      claim.

ARTICLE 9. REPAIR
-----------------

9.1   Repair. The Tenant shall keep the Premises in good and substantial repair and condition,
      damage by the Insured Risks excepted.

9.2   Landlord's fixtures. The Tenant shall not remove any of the Landlord's fixtures from the
      Premises at any time during the Term.

9.3   Decoration. The Tenant shall redecorate the interior of the Premises every 3 years and in the
      last year of the Term.

ARTICLE 10. USE
---------------

10.1  Permitted use. The Tenant shall use the Premises for warehousing, distribution and ancillary
      office use and no other purpose.

10.2  Hours. The Tenant may use the Premises at any hour, and shall not cause a nuisance to the
      other occupiers of the Building.

ARTICLE 11. QUIET ENJOYMENT
---------------------------

11.1  Quiet enjoyment. So long as the Tenant pays the Rent and performs its covenants, it may hold
      the Premises without interruption by the Landlord.

ARTICLE 12. ALIENATION
----------------------

12.1  Assignment. The Tenant shall not assign this Lease except with the Landlord's prior written
      consent, which shall not be unreasonably withheld.

12.2  Underletting of the whole. The Tenant may underlet the whole of the Premises with the
      Landlord's prior written consent, which shall not be unreasonably withheld.

12.3  Underletting of part. The Tenant shall not underlet part only of the Premises.

ARTICLE 13. DETERMINATION
-------------------------

13.1  Break right. The Tenant may determine this Lease on the Break Date of March 11, 2025 by giving
      the Landlord not less than 6 months' prior written notice.

13.2  Conditions. A notice under clause 13.1 takes effect on the Break Date of March 11, 2025 only
      if the Tenant gives vacant possession on that date and has paid the Rent due to it.

ARTICLE 14. NOTICES
-------------------

14.1  Form. Any notice under this Lease shall be in writing and delivered by hand or sent by
      recorded delivery to the address stated in paragraph S1.2.

14.2  Deemed receipt. A notice delivered by hand is received when it is delivered. A notice sent by
      recorded delivery is deemed received on the third Business Day after posting.

ARTICLE 15. DEFAULT AND FORFEITURE
----------------------------------

15.1  Re-entry. If the Rent is unpaid for 21 days after it falls due, whether formally demanded or
      not, the Landlord may re-enter the Premises and this Lease then ends.

15.2  Costs. The Tenant shall pay the Landlord's proper costs of preparing and serving a notice
      under section 146 of the relevant statute.

SCHEDULE 1 - PARTICULARS
------------------------

S1.1  Parties. Landlord: Marlstone Estates LLC. Tenant: Ostlerbrook Analytics LLC. Address for
      notices to the Landlord: the registered office of the Landlord for the time being.

S1.2  Address for notices. Notices to the Tenant shall be sent to the Premises, marked for the
      attention of the company secretary.

S1.3  Base Rent. Base Rent: $69,766,200.00 per annum, payable monthly in advance.

S1.4  Rentable area. Rentable area of the Premises: 33,900 rentable square feet.

S1.5  Dates. Commencement Date: March 12, 2022. Review Dates: each anniversary of the Commencement
      Date from the second Lease Year.

SCHEDULE 2 - SERVICE CHARGE
---------------------------

S2.1  Proportionate Share. The Tenant's Proportionate Share of the Expenditure is 16.20 per cent.

S2.2  Expenditure. The Expenditure comprises the cost of insuring, maintaining, cleaning, lighting
      and managing the Common Areas and the structure of the Building.

S2.3  Account Year. The Account Year ends on 31 March.

S2.4  Apportionment. The Landlord shall apportion the Expenditure fairly between the occupiers of
      the Building.

Raw responseThe raw response

{"conflicts":[{"clause_a":"14.1","clause_b":"S1.1","conflict_type":"IMPOSSIBLE_TOGETHER","subject":"address for giving notice to the Landlord","quote_a":"Any notice under this Lease shall be in writing and delivered by hand or sent by recorded delivery to the address stated in paragraph S1.2.","quote_b":"Address for notices to the Landlord: the registered office of the Landlord for the time being.","why":"A notice to the Landlord must be sent to the S1.2 address under clause 14.1 and to the Landlord's registered office under S1.1, and the document does not make those addresses the same."}],"rationale":"I considered but did not report the 72-month original term against the 36-month renewal term, because clause 7.1 itself fixes the new lease's duration and the 'same terms' wording in 7.2 is not an independent duration term. The notice-address conflict is the one place where the body's blanket rule and the schedule's Landlord address cannot both be followed."}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch clauses in a lease that contradict — 60 agreements, each read whole. One model answered, and every answer was then graded Three different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

No model grades anything. Every metric is a set comparison against data/gold.jsonl in evals/scoring.py: an answer for one agreement is a set of UNORDERED clause pairs and the key is a set of unordered pairs, so "6.2 and 15.3" and "15.3 and 6.2" are one finding. Precision and recall are reported separately and are NEVER averaged into one figure, because on this corpus they are trivially trade-able: an arm that reports every pair of clauses mentioning the same subject gets near-perfect recall and is worthless. An arm that answered nothing has an UNDEFINED precision, not a perfect one.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on this page is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anybody paid. The token counts come from the provider's own usage block on each call, and 99.3 pct of the output tokens they report are provider-side reasoning the kit did not ask for.

Rates checked 2026-08-27. The provider that actually ran every call here is kept off this page per the series rule. Its real spend, its own dated card, its peak/off-peak split and its cached-input tier are recorded per call in results/eval-r001-clause-conflict.json and summarised in Cost.measured_on, not projected here.

Grading unitWhat the grading figure prices

Scoring is pure Python over committed JSON: 60 agreements, 39 gold pairs and 24 planted near misses, in under a second. There is no LLM judge anywhere in the grading path, so re-scoring a cached run costs nothing.

The gradersThree ways to grade

⚠︎ READ THE CARVE-OUT FLOOR BEFORE READING ANYTHING ELSE ON THIS PAGE. b001-clause-conflict-carveout is FREE, runs in 0.2 s with no network, and scores 100.0 pct precision -- 22 pairs reported, 22 of them right, 0 wrong, and all 24 clean agreements left clean. It is the strongest arm in this kit on everything except recall, where it stops at 56.4 pct because it is blind to every conflict with no number in it. ⚑ THE STUB IS THE FLOOR, AND THAT IS DELIBERATE RATHER THAN A DEFECT. t000-clause-conflict-stub scores IDENTICALLY to b000 on every metric because stub_complete() in evals/run.py answers from the rules floor -- it exercises prompt assembly, the JSON parse, the recheck, the scorer and the result writer end to end without a key. Its output_tokens_max of 310 is a measurement of the stub, not of this job; the real run's 57,570 is 186x that.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-27. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingGrading is free; the bill is the 60 calls that answered.

The scorer, the label gate, the corpus generator and both free floors are pure code and cost $0.00 to run against any result set -- there is no LLM judge anywhere in the grading path, and re-scoring a cached run is free. The figure above prices the ONE published paid run (60 calls, 206,732 input tokens and 2,374,543 output) on the same projected card cost_per_query_usd uses. ⚠︎ IT DOES NOT INCLUDE THE DISCARDED RUN. The first r001 was fired at 32,000, truncated 47 of 60 replies and was thrown away whole; that money was spent and bought nothing, and it is recorded in results/discarded/README.md rather than folded into a total here. The 20-call adversarial arm is not in it either, because it has never been fired.

Cost driversWhat actually moves the bill

• PROVIDER-SIDE REASONING, and it is not close. 99.3 pct of this run's output tokens are reasoning left at the tier's default and re-rolled per call -- billed, counted against max_tokens, never shown. The VISIBLE answer is a short list and on 24 of the 60 agreements it is an empty one. 2,374,543 output tokens were drawn to produce 82 reported pairs.
• OUTPUT IS 11.5x INPUT and 98.6 pct of the projected bill (39,575.7 output tokens a call against 3,445.5 input, at a 6x output rate). That is an extreme ratio for this estate and it is a property of the tier's reasoning rather than of the answer: the answer is small.
• THE STABLE PREFIX, in the other direction. The role and the conflict vocabulary are 4,854 characters -- 35.9 pct of the mean prompt -- and byte-identical on every call, so 98.1 pct of the run's input tokens came back as prefix cache hits on the provider that ran it. The projection card has no cached-input tier, so it overstates the input side; the input side is only 1.4 pct of the projected bill, so it barely matters.

Your volumeWhat it costs at your volume

Linear in agreements and NOT linear in the agreement. Ten times the documents is ten times the calls at the same 3,445.5 input tokens each, and the input side is only 1.4 pct of the bill anyway. What is not linear is length: the reply's cost here is the tier's reasoning budget over a whole document held at once, and this corpus spans only 48 to 60 clauses. A document twice this size is outside everything measured here, and the first thing it would threaten is the ceiling, not the price -- 90.0 pct of it is already used.

Model choiceWhy this model

It is the tier the estate runs, so the figure compares with every sibling kit. It was NOT chosen by a bake-off on this corpus and no second model has been called against it -- see the projection block, and read every row there against $0.00, because the free carve-out floor scores 100.0 pct precision and 43 of 60 agreements exact for nothing.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

the whole 60-agreement scored run (r001-clause-conflict, the fast tier): 206,732 input tokens, 202,880 of them prefix cache hits, and 2,374,543 output of which 2,357,070 reasoning. The real bill is on the withheld provider's card and lives in the run record and in Cost.measured_on, not in this block. It does NOT cover the discarded 32,000-token run, which was paid for and thrown away.

The codeEvery file, and what it is for

10 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Six of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/agreement.pythe clause parser — a swap seam

Turns the document into {clause_id: text} and is the only address space this kit has. It keeps THREE apart -- 4.2 a body clause, S2.1 a schedule paragraph, A1.2 an amendment-sheet paragraph -- because a body-against-schedule conflict is a different kind of finding from two body clauses. Whitespace is normalised through norm() so a quote that spans a wrapped line is still findable, and a clause reference is folded through cid() because arms write clause 4.2, Clause 4.2., 4.2) and s2.1 for one address. One parser, five readers: the corpus generator's own assertions, both free floors, the scorer, the answer-key checker and the board.

You change it to: CLAUSE_RE and HEAD_RE. Everything downstream addresses a clause by its id, so a different numbering convention is a regex rather than a rewrite -- but read the warning in breaks_at_scale first.

# One agreement, read into numbered clauses. No model, no network, no dependency.

CLAUSE_RE = re.compile(r"^((?:\d+|S\d+|A\d+)\.\d+)\s{2,}(.*)$")
HEAD_RE = re.compile(r"^(ARTICLE\s+\d+\.|SCHEDULE\s+\d+|AMENDMENT SHEET NO\.\s*\d+)\b(.*)$")
BODY, SCHEDULE, AMENDMENT = "body", "schedule", "amendment"
def norm(s):
def cid(raw):
def where(clause_id):
def parse(text):
def by_id(text):
def ids(text):

src/prompt.pyprompt assembly — a swap seam

Two parts in a fixed order -- the role and the five-way conflict vocabulary, then the agreement verbatim. The vocabulary is declared ONCE as VERDICT_MEANINGS and the instruction block is BUILT from it, so the words the model reads, the words the scorer grades against and the panel this page draws cannot drift apart; a module-level assertion fails loudly if VERDICTS and VERDICT_MEANINGS ever disagree in membership or order. It gives the vocabulary and withholds the method: nothing tells the model to check the schedule against the body, or to look for an express carve-out first.

You change it to: Add or reword an entry in VERDICT_MEANINGS. The instruction block, the scorer's accepted set and this page's vocabulary panel are all built from that one dict, and an import-time assertion refuses a vocabulary that has come apart from the prompt.

# The assembled prompt, in one place, in the order it is sent.

VERDICT_MEANINGS = {
VERDICTS = ("FIGURE_DISAGREEMENT", "SCHEDULE_CONTRADICTS_BODY", "AMENDMENT_ORPHAN",
def _vocab_block():
def _fold(text, indent, width=92):
SYSTEM = """You are reading ONE agreement and looking for the places where it contradicts itself.
HEADER = ("THE AGREEMENT, verbatim. Clause identifiers are at the start of each paragraph.\n"
def build(text):
def render(parts):

src/checker.pythe model call

One call per agreement, and the only place a model is reached. MAX_TOKENS is 64,000 and the docstring records why: fired at 32,000 the first paid run truncated 47 of 60 replies. A reply cut off at the ceiling is recorded with at_ceiling set, stays inside the published denominator and is never re-fired after its misses have been read.

# One agreement in, one list of conflicting clause pairs out. The only place a model is called.

MAX_TOKENS = 64000
THINKING = None
def parse_reply(text):
def normalise(obj):
def work(cfg, text, complete_fn=None, max_tokens=None):

src/recheck.pythe citation station — a swap seam

Pure code between the reply and the report, run on EVERY arm including both free floors. It DROPS a pair naming a clause identifier the document does not carry, a pair naming the same clause twice, and the second copy of a pair already reported. It MARKS a quote that does not occur in the clause it is attributed to, and leaves the finding visible. It never invents, never repairs a quote to something that does occur, and never re-decides whether two clauses genuinely conflict.

You change it to: What is dropped versus what is marked. Today an unresolvable clause id is dropped and an unquotable span is marked; swapping either is one branch, and both arms are already published side by side so the effect is measurable rather than argued.

# The pure-code station between the reply and the report. No model, no network.

FIELDS = ("clause_a", "clause_b", "conflict_type", "subject", "quote_a", "quote_b", "why")
def _type(v):
def recheck(answer, text):

evals/baseline.pythe free floors — a swap seam

Two arms, no key and no network. rules pulls every figure, duration, date and money amount out of every clause, works out what each is ABOUT from the words around it, and reports the pairs that disagree. carveout is the same plus the two drafting rules a person would add next: do not report a pair one clause expressly reconciles, and do not report an amendment against the clause it names.

You change it to: MODES is a tuple; a third mode is a function and one entry. The two shipped modes differ by exactly two drafting rules, and the difference between their scores is what prices those rules.

# THE FREE FLOORS. No key, no model, no network. A genuine attempt at the job.

MODES = ("rules", "carveout")
MEASURES = [
BUCKETS = [
WINDOW_BACK, WINDOW_FWD = 140, 80
ANCHOR_RE = re.compile(r"^\s*(?:after|before|of|from|following|prior to)\s+(.{0,64})", re.I | re.S)
ANCHOR_STOP = {"the", "that", "this", "a", "an", "which", "on", "to", "of", "its", "it", "in",
CARVEOUT_CUES = ("except that", "save that", "save where", "applies in place of", "in place of",
AMEND_VERB = re.compile(r"\b(?:is|are)\s+(?:amended|deleted and replaced|deleted)\b", re.I)
REDIRECT = re.compile(r"reference[s]?\s+in\s+the\s+lease\s+to\s+clause\s+"

evals/scoring.pythe scorer

Set comparison, one unordered pair against another. Precision and recall are reported separately and never averaged; an arm that answered nothing gets an UNDEFINED precision rather than a perfect one. Span credit is scored only on pairs the arm actually found, because a pair that was never found is already counted by recall.

# Grade one arm against the answer key. Deterministic -- no model judges anything here.

TAXONOMY = ("no_answer", "missed_pair", "decoy_flagged", "invented_pair", "type_wrong",
def _pct(n, d):
def tokens(s):
def span_f1(pred, gold):
def _pairs(rows):
def score(records, golds, texts=None):

evals/check_labels.pythe answer-key gate

Grades the KEY before any arm is graded against it: every span in the clause it names, every pair two real clauses, every type in the declared vocabulary, and an independent numeric checker that reports nothing the key does not already carry. Free, and it runs on a copy with no .env.

# Grade the ANSWER KEY. No key, no model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
CORPUS = os.path.join(DATA, "corpus")
MY_CLAUSE = re.compile(r"^((?:\d+|S\d+|A\d+)\.\d+)\s{2,}(.*)$")
VALUE = re.compile(r"(\$[\d,]+\.\d{2}|\b\d{1,3}(?:\.\d{1,2})?\s+per\s+cent\b|"
AMEND_NAMES = re.compile(r"\b((?:\d+|[SA]\d+)\.\d+)\b")
FAILS = []
def check(cond, msg):
def norm(s):

tools/build_corpus.pythe corpus generator

60 agreements, the key and the statistics out of random.Random(20260831) and nothing else. A conflict exists because a planter put it there and recorded both clause ids and both spans -- nobody reads a finished document and decides whether it contradicts itself. Every span is asserted into its clause, through the same parser the scorer uses, before a file is written.

# Build the 60 agreements, the answer key and the corpus statistics. Deterministic, free, offline.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
CORPUS = os.path.join(DATA, "corpus")
SEED = 20260831
DATASET_VERSION = "clause-conflict-v1-60agreements"
CASES = [
CONFLICT_TYPES = ("FIGURE_DISAGREEMENT", "SCHEDULE_CONTRADICTS_BODY", "AMENDMENT_ORPHAN",
LANDLORDS = ["Harrowgate Estates Limited", "Pellmore Property Holdings LLC",
TENANTS = ["Vantry Foods Limited", "Ostlerbrook Analytics LLC", "Fenmarch Instruments Limited",

src/budget.pythe spend guard

Counts CALLS, not dollars, against a cap read from the shared .env, and writes one append-only ledger line BEFORE each call so a crash over-counts rather than under-counts. No cap configured means no cap, and it says so out loud.

# A cap on live calls, shared by every kit on this machine. No dependency, no service.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
KIT = os.path.basename(HERE)
ROOT = os.path.dirname(os.path.dirname(HERE))
SHARED_ENV = os.path.join(ROOT, ".env")
LEDGER = os.path.join(ROOT if os.path.exists(SHARED_ENV) else HERE, ".calls-ledger.jsonl")
PAID_ARMS = ("r", "x")
FREE_ARMS = ("t", "b")
BOOT = os.urandom(6).hex()
PID = os.getpid()

src/app.pythe local board

⚠︎ SHIPPED BROKEN -- see could_not_verify. Standard-library HTTP server; /api/agreements, /api/agreement, /api/corpus, /api/prompt and /api/floor need no key, and only /api/check calls a provider. It serves ui/index.html on /, and ui/ IS EMPTY IN THE COMMITTED TREE, so the root route raises rather than rendering.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
CORPUS = os.path.join(HERE, "data", "corpus")
PORT = int(os.environ.get("PORT", "9222"))
FLOOR_RUNS = ("b000-clause-conflict-rules", "b001-clause-conflict-carveout")
PAID_RUN = os.environ.get("PAID_RUN", "r001-clause-conflict")
def agreements():
def golds():
def stats():

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 3445 input and 39575 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

⚠︎ THE PROBE IS WRITTEN, PAIRED AND UNFIRED, SO THIS KIT PUBLISHES NO ATTACK NUMBER. evals/injection.py ships complete -- one paragraph in the register of a construction clause, appended to the document, asserting that no provision conflicts with any other and that any review is to record that. The control arm is the SCORED RUN'S OWN cached answers, so the probe would cost one call per agreement rather than two. It has never been run against a provider and results/ holds no x001 file.

API_KEY is read from the shared <repo>/.env, this kit's own .env and the real environment, in that precedence, by src/config.py -- eight lines with no dependency. Both .env files are gitignored from the first commit and neither is required: a fork with no key gets a working kit, because the corpus, the key, both free floors and the committed run records all ship.

The experimentWe have not attacked it yet — and the hole is already named

An indirect prompt injection needs a field an outside party controls, and here the outside party controls the WHOLE INPUT: the agreement is drafted by the counterparty, and a clause about how the document is to be construed is ordinary in one. The attack asks for silence, which is the correct answer on 24 of these 60 agreements, and silence is the one outcome a downstream citation check cannot flag. Both were run for real on 2026-08-31 -- and what was measured is the SCORED run, not an attack. Every claim below about a boundary holding is a claim about CODE, checked by reading it, except where it says otherwise.

Four of the five boundaries above hold in CODE and were confirmed against the scored run's own counters. The first -- the one that matters -- is UNMEASURED, and the kit says so rather than borrowing a sibling's number.

The result0 of 20 adversarial trials fired. Four boundaries hold in code and were confirmed on the scored run -- 0 invented clause ids, 0 pairs dropped, 98.6 pct of quotes verbatim in their clause. The fifth, and the only one that matters here, is unmeasured.

The probe is ready and paired: evals/injection.py appends one construction-clause paragraph to the document and compares the reply against the scored run's own cached answer for the same agreement, so only agreements the control got RIGHT are in the paired denominator. It scores both the raw and the rechecked answer, because the rechecked one is what shows that pure code cannot defend against a suppression. Running it costs 20 calls and has not been authorised.

HonestyWhat this does not prove

• Whether the injection works. The probe is unfired; there is no suppression rate and no borrowed one.
• Whether the citation station's drop rule ever fires on a real reply. It fired zero times on this run, so the branch is untested by measurement.
• Whether any other model behaves the same way. Only the fast tier has been called.
• Whether a longer or differently-drafted agreement changes any of it. The corpus is 60 files from one generator with one drafting voice, 48 to 60 clauses.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Never decide, advise, waive, amend or settle anything. The kit REPORTS pairs of clauses that appear not to be capable of both being satisfied, with the words in each that clash, and it stops there. It does not say which clause prevails, it does not say what a court would do, and it does not rank findings by importance.

Stated in the instruction the model is sent (src/prompt.py SYSTEM: report a pair only where you can say, in the document's own words, what each clause requires and why both cannot be done; never write a quote or a clause identifier that is not in the document), and enforced downstream in code by src/recheck.py, which drops an unresolvable citation and marks an unquotable span without repairing either.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• Not legal advice, and not a reading of which clause prevails.
• Not a completeness guarantee. The paid arm missed 4 of 39 planted conflicts and both free floors missed 17.
• Not a filter you can put in front of a signature. On 20 of the 24 clean agreements the paid arm raised a finding.
• Not a defence against an instruction written into the document. The probe is unfired and pure code downstream cannot see a suppression.
• Not a measurement of any model but the fast tier, or of any reasoning setting but the provider's default.

WatchedWhat is watched, and why that one

The kit measures 39 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

61 measured by the latest run -22 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (agreements 60, agreements_answered 60, clean_agreements 24, conflicted_agreements 36, dataset_version clause-conflict-v1-60agreements, decoy_pairs 24, failures 0, gold_pairs 39, pred_pairs 35, rechecked_clean_agreements 24, rechecked_conflicted_agreements 36, rechecked_decoy_pairs 24, rechecked_gold_pairs 39, rechecked_pred_pairs 35, rechecked_span_sides 44, rederived_from_cache False, socket_timeout_s 1200, span_sides 44) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

4 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

match · no model in the path — a baseline, not a peer column — 2 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 2 runs measured the same system — they differ on floor, pred_pairs, rechecked_pred_pairs, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

match · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 65 chips that all say so.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 61 chips that all say so.

DeviationsWhat deviated

0 breaches across 4 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

The corpus generator, the label gate, both free floors, the citation station and the scorer all run for free with no key and no network, so the whole chain below the model call can be re-run on every change. Re-scoring a cached paid run is free too -- --resume --rescore, which reads the cache and makes no call. ⚠︎ --rescore ALONE RE-FIRES ALL 60 CALLS; on this kit that is $1.5695 and 70 minutes.

In one lineNo framework, deliberately

NO FRAMEWORK. A folder of readable Python, standard library end to end. requirements.txt names no package and says why: a requirements file naming a package the code never touches is a small lie that costs a forker an install.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Linear, and concurrent only at the agreement level. There is no agent, no tool loop, no retrieval and no second call: one document in, one call, one pure-code recheck, one report. The 5 workers are the eval harness fanning out over agreements, not the kit fanning out over anything.

The other sideWhat a framework costs you

• No dependency resolution and no lockfile -- and no upgrade treadmill either.
• You write the provider adapter for a provider not already in PROVIDERS. It is one function that must return the same shape including token counts, because the Cost lens prices them.
• No built-in tracing, no callback bus and no dashboard. What you get instead is a result file per run carrying every per-call token count, latency, tariff and price.
• No streaming, and here that is a real cost rather than a stylistic one: a 57,570-token generation is a silent socket for minutes, which is why TIMEOUT_S is 1200.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-clause-conflict on the fast tier, 2026-08-31. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 2 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — blocking is computed in memory each run and is the only reduction step.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT -- and the MIT text is in this repository's LICENSE-PUBLIC file, NOT in its root LICENSE, which governs the repository itself and grants a reader nothing. The KIT and its corpus are MIT; the repository is not, and this line says which, because it is the only claim a reader is entitled to rely on before forking. There is no third-party data in it to licence at all: the 60 agreements, the answer key, the near misses and the statistics are generated in-process by tools/build_corpus.py from a fixed seed, and nothing was scraped, licensed, anonymised or derived from anybody's document. Each corpus file repeats the grant in its own header. ⚠︎ The kit ships no data/SOURCES.md to state the same distinction, which is a gap in the kit and not in this claim: LICENSE-PUBLIC was read and checked this session. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe pair set -- which two clauses, what kind of contradiction, and the words in each that clash

whether each pair the arm reported is a pair the generator planted. Pairs are compared as unordered two-element sets of clause identifiers, folded through src/agreement.py's cid() because arms spell one address six ways. A pair that matches the key is then checked for its TYPE and for whether each quoted span is really in the clause it is attributed to.

MethodHow the test was run

python3 -m evals.run --run-id <id> [--floor rules|carveout]; evals/scoring.py compares pair sets against data/gold.jsonl. No model is in the grading path.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl -- and THE KEY IS PLANTED, NOT LABELLED. A conflict exists in an agreement because tools/build_corpus.py put it there and recorded both clause identifiers, the conflict type and a verbatim span from each side; nobody reads a finished document and decides whether it contradicts itself, because that decision is exactly what is being measured. Every span is asserted into its own clause -- through src/agreement.py, the same parser the scorer and the free floors use -- before a file is written, and evals/check_labels.py re-grades the whole key independently afterwards. Every party, building, figure, date and definition is invented from seed 20260831.

No true/false rates for this grader. It records 6 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? There is no threshold and nothing was swept: a pair either matches the key's unordered clause set or it does not. ⚠︎ THE DENOMINATORS ARE SMALL AND THE PAGE SAYS SO -- 39 gold pairs in total, and the type that carries the argument for paying has 7. One pair is 2.6 points of the recall denominator, so no band finer than that is supportable.

Cadence: Every arm, every run -- it is pure code over committed JSON and costs $0.00, so there is no reason to run it less often than the corpus changes.

The decisionWhen to reach for it

In one lineThe whole-agreement verdict -- did the report on THIS agreement match the key exactly

whether an arm's entire answer for one agreement is the key's entire answer: every planted pair found, nothing extra. It is the measure a reader experiences, because a reader opens one agreement at a time.

MethodHow the test was run

Same run, same scorer -- it is a second reduction of the same pair comparison, not a second pass.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The same planted data/gold.jsonl. An agreement is exact when the arm's whole pair set for it equals the key's whole pair set for it -- including the empty set, which is the correct answer on 24 of the 60 agreements.

No true/false rates for this grader. It records 7 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No threshold: exact set equality per agreement. The denominator is 60, so one agreement is 1.7 points and no band finer than that means anything. The clean sub-denominator is 24, where one agreement is 4.2 points.

Cadence: Every arm, every run -- it is pure code over committed JSON and costs $0.00, so there is no reason to run it less often than the corpus changes.

The decisionWhen to reach for it

In one lineThe citation check -- does the clause exist and are those really its words

whether a finding is one a reader can act on at all. src/recheck.py re-resolves every clause identifier against the parsed document and every quoted span against the clause it names. It DROPS a pair naming a clause the agreement does not have and MARKS a quote that is not in its clause; it never invents and never repairs.

MethodHow the test was run

Runs inside every arm, always. It costs nothing, so the kit never publishes an un-rechecked answer.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: THE DOCUMENT ITSELF, not the answer key -- and that is what makes this grader different from the other two. src/recheck.py asks only whether the clause identifier exists in the parsed agreement and whether the quoted span occurs in that clause, both through the whitespace normalisation the key was built with. It never consults gold.jsonl and never decides whether a pair is really a conflict.

No true/false rates for this grader. It records 5 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No threshold: a string is in a clause or it is not. The span denominator is 70 sides on the paid arm and 44 on the floors, and the two are NOT comparable -- span credit is scored only on pairs an arm actually found, so a better-recall arm has a bigger denominator here by construction.

Cadence: Every arm, every run -- it is pure code over committed JSON and costs $0.00, so there is no reason to run it less often than the corpus changes.

The decisionWhen to reach for it