The business caseThe problem this solves
A customer is contracted for a set of entitlements and their tenant is provisioned with another set, and the two drift apart quietly. Over-provisioning is revenue nobody is billing; under-provisioning is service somebody paid for and is not getting; a contracted line absent from the export altogether is both at once. The reconciliation everybody runs is a two-column diff between the order form and the provisioning export, and it is wrong in a way that looks right -- because THE CONTRACTED QUANTITY IS NOT A FIELD ANYWHERE IN THE PACK. It is the order form's base plus or minus whatever chain of amendments is in force on the reconciliation date, and every word there does work: an amendment signed in June and effective next January has changed nothing yet; a restatement replaces the base where an addition adds to it; a later amendment can rescind an earlier one, so the biggest and latest number on the page is the one that does not apply; a quantity may be a percentage of the quantity then in effect, a figure written out in words, or a ramp table whose tier you pick by date; a co-termination moves a count onto another line and zeroes this one; and the order form and the export use different names for the same product, joined only by a footnote in the catalog. Someone opening an account's contract folder before a renewal or a true-up: reading the order form, then every amendment in date order, working out what each one did to which line and whether it was in force yet, joining that to a provisioning export keyed by codes the contract never uses, and writing the difference down.
Audience
Revenue operations, deal desk and billing teams who reconcile provisioned entitlements against the contract stack before a renewal, a true-up or an audit -- and the people who build tooling for them. Every number on these pages came from one real run of this code, not from a vendor page.
The inputThe actual customer account packs
The corpus is 32 customer account packs, 0.13 MB (txt 32). A real entitlement pack names a named customer, the money they pay, the seats their staff hold and an account owner's judgement about whether to bill them for the difference -- so it is not publishable, and a kit whose corpus cannot be published is a kit nobody can re-run, which would make every figure on this page an assertion rather than a measurement. Generating it also buys the one thing a scraped corpus cannot give: an answer key built from the same structures as the prose rather than parsed back out of it, so it cannot merely prove the generator's regex agrees with itself.
The corpus
- The 32 customer account packsgenerated from a fixed seed, so no real record, person or institution appears in it.
- Where each came fromwritten for this kit rather than collected — the corpus is generated in the kit's own repository, so there is no third-party data in it.
Swap this folder for your own material and the kit is pointed at your customer account packs. That is the whole change — there is no database to migrate.
ACCOUNT
=======
Account: NWL-6305 Northwind Logistics Ltd
Reconciliation as of: 2026-08-27
Prepared by: Revenue Operations (entitlement reconciliation)
ORDER FORM
==========
Order Form OF-2893, executed 2025-06-19, effective 2026-01-01, initial term 36 months.
Line Product (as named in this Order Form) Quantity Unit
1 Document Storage 8,500 GB
2 Core Platform Users 178 seats
3 Directory Sync and SSO 826 seats
4 Enablement Credits 15 credits
5 Sandbox Environments 1 environments
6 Production Environments 20 environments
General terms. Quantities stated above are the quantities in force on the effective date
of this Order Form and remain in force until amended in writing. An amendment takes effect on
its stated Effective Date, which may differ from the date it was signed.
AMENDMENTS
==========
Amendment 1 to Order Form OF-2893
Signed: 2026-06-11
Effective: 2027-01-01
Effective on the Effective Date set out above, the quantity of Enablement Credits in Section 1 is amended and restated in its entirety to thirty (30) credits. For the avoidance of doubt no change to the quantity takes effect before that date.
Amendment 2 to Order Form OF-2893
Signed: 2026-03-25
Effective: 2026-04-01
The Production Environments quantity then in effect is increased by twenty-five percent (25%) as of the Effective Date. No other Section is amended.
PRODUCT CATALOG
===============
Provisioning key Contract name used on the Order Form NoteAbridged — the file continues.
The outcomeWhat a good result looks like
A worksheet: one row per provisioning key, each carrying the quantity contracted on the reconciliation date -- a number, or UNLIMITED, or NONE -- and one of MATCHED, OVER_ENTITLED, UNDER_ENTITLED or NOT_PROVISIONED, with the document that establishes it named. It is read and signed by a person; nothing here bills, provisions or deprovisions anything.
And when it cannot
⚠︎ THE PAID ARM MADE NO ERRORS AT ALL ON THIS CORPUS -- 257 of 257 cells, both fields right on every one -- AND THAT IS A FINDING ABOUT THE CORPUS, NOT A RESULT TO BE PROUD OF. The hard classes here were built to defeat CODE: a percentage instead of a figure, a quantity written out in words, a rescission, a ramp table, a co-termination. Every one of them is trivial for a competent reader, and a frontier model is a competent reader. The corpus therefore has no ceiling above the task: it cannot separate this tier from a better one, and it cannot show what this arm would get wrong. Diagnosed and NOT fixed -- regenerating a harder corpus after reading the result would be fitting the test to the answer. THE FAILURES ON THIS PAGE ARE THE FREE FLOORS', and they are real: the strongest one misses 44 of 257 cells, and on the 38 rows where the operative sentence genuinely decides the quantity it scores 0.
Where it fitsWhat did work
Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.
- Every account on your book has one order form, no amendments, and a provisioning export keyed by the same names the paper uses. — the free floor -- mirror the order form, $0.00
It scores 100.0 pct on the 100 rows the two tables decide and costs nothing. There is no chain to walk. - Your amendments are templated -- 'amended and restated in its entirety to N', 'amended to add N additional' -- and they carry a stated Effective date. — the free floor -- dated ledger, $0.00
It takes 100.0 pct of the dated band and 82.88 pct overall for nothing, and it is deterministic, instant and auditable line by line. - Your amendments are negotiated prose: percentages, ramps, rescissions, co-terminations, quantities spelled out. — the model, at $0.02522258 per account
It took 38 of 38 of exactly those rows, against the strongest free floor's 0, and raised zero false findings on 124 agreeing lines. - You need the same answer twice. — the free floor, whichever one fits
It is deterministic. The model arm was run once and no repeat was taken, so nothing on this page separates a right answer from a lucky one.
At a glanceHow the whole thing runs
Run once, for real, on 2026-08-27. Every figure on these pages was captured from that run — nothing is written from intent.
14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.
Should you use this?What you bring, where it stops, and when not to use it
Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.
| What do I have to bring? | tools/build_corpus.py writes data/corpus/*.txt and data/gold.jsonl. NONE OF THE MEASURED FIGURES ON THIS PAGE TRANSFER TO YOUR OWN ACCOUNTS. Corpus lens → |
| When is this the wrong choice? | Avoid: Do not use it anywhere an amendment exists: it scores 43.33 pct on the 90 rows an effective date decides. That is the case against the best-fitting scenario (“Every account on your book has one order form, no amendments, and a provisioning export keyed by the same names the paper uses.”). 4 scenarios scored in all, each with its own. Eval lens → |
| Where does it stop working? | An order form that is not a fixed-width table. src/pack.py's four regular expressions parse this generator's output exactly and will parse nothing else without changing. 5 recorded failure modes, each from a run rather than a guess. Corpus lens → |
| What was never verified? | WHETHER THIS CORPUS IS HARD ENOUGH TO MEASURE THE ARM IT SCORED. It is not. 5 items this kit says it could not check. Eval lens → |
| Can I run this on a model I control? | Yes — any OpenAI-compatible endpoint, including one on your own hardware. The shipped adapter takes its host from BASE_URL and its model from MODEL, so nothing in src/ changes. The published figures come from 2 models on the fast tier and the deliberating tier, one provider, one key. Prompt lens → |
| And if it fits — what do I stand up? | 4 artifacts with a stated home and a stated egress, and 4 decisions each with what you provision past its ceiling — plus what was not measured. That is the next page, not this one. step 14 — Run it in your environment → |
Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).
Last verified 2026-08-27 — r001-entitlement-drift. Every figure on these pages was captured from that run.
Run itHow this reaches your data
Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.
Run this on your own data
- The pipeline, its eval harness and the runs behind every numberdeployed inside your environment, on your own model endpoints, against your own documents.
- The corpus above is the shape, not the limitit is a folder swap, and there is no database to migrate.
Checked before this shipped — Every pack, the answer key, all three free floors, the recorded runs, the injection probe's result and the whole UI work with no key and no network. A key unlocks exactly one thing: the 'Run it live' button, which makes one call for the account on screen.
