A title can be licensed to many companies at once, and most overlaps are fine. This app checks each pair of deals against the contract's own rules and flags the ones that actually clash.
PresenterOpens the private repo. Visible to admins only.
For a rights managerMedia & Entertainment · Legal Services
Why it matters
Today's manual process, and the same job with the app
A rights manager at a film or TV licensor, clearing new deals against everything already signed.
✕Today's manual process
1Pull up both contracts and compare territory, platform, language and exclusivity line by line.
2Check the calendar for any days the two license windows share.
3Reread every clause for a holdback, a carve-out or a wider definition that changes the answer.
4One missed clause and a partner airs or streams a title it was never cleared for.
Every pair read and judged manually
✓With the app
1Both deals load together every term lined up, side by side.
2The app reads the clauses including holdbacks and carve-outs the calendar can't see.
3It gives a verdict conflict or fine, with the exact clause behind it.
4Real conflicts go to a lawyer before anything is licensed, aired or changed.
Every verdict comes with its clause
See it work
One real case, worked step by step
Greyling Networks' window on The Burning Meridian ends October 2027; Delphi Screens starts fifteen days later, but a holdback still blocks 46 days.
Check whether two license deals on one show clashReference appBuilt to be shaped to your process
5
1Grant A's terms Exclusive rights in Mexico, on EST, through October 2027.
2Grant B's terms Same market and platform, starting fifteen days after Grant A ends.
3The app's read The windows never touch, but a holdback still makes this a conflict.
4The proof Clause A-4.2 extends Grant A's exclusivity 46 days into Grant B's window.
5The outcome Sent to a rights lawyer; no grant is changed automatically.
For engineers
How it is built, and how we measured it
All fourteen steps of the build are written up, from the business case to running it in your own environment.
A small, forkable project that does one job end to end. Run once for real, and every figure on these pages captured from that run.
PresenterOpens the private repo. Visible to admins only.
The business caseThe problem this solves
A title is licensed to several parties across territories, windows, languages, platforms and exclusivity levels, and somebody has to decide pair by pair whether two grants can both be honoured. Most apparent overlaps are FINE -- non-exclusive grants may coexist, a territory may nest and still be carved out by name, a full overlap may breach nothing. And some pairs that share not one day on a calendar are genuine conflicts, because an exclusivity runs past its own window under a holdback, or one contract's definition of a platform swallows a service the other names. The clause decides, not the calendar. a rights conflict report that interval-overlaps two date ranges and flags every intersection -- which on this corpus invents 34 conflicts that are not there and walks past 38 that are.
Audience
a rights manager working a clearance list, and a head of business affairs deciding whether the conflict report is finding anything. Every number on these pages came from one real run of this code, not from a vendor page.
The inputThe actual clearance requests
The corpus is 120 clearance requests, 0.83 MB (txt 120). A distributor's rights ledger gives who bought what, where, for how long, at what price and -- crucially -- which grants are exclusive. There is no public one. A scrubbed export is worse rather than better: the clause text is exactly what makes this task hard, and it is the first thing a scrubber removes -- or it was never in the ledger at all, because it lives in the contract.
The corpus
The 120 clearance requestsgenerated from a fixed seed, so no real record, person or institution appears in it.
Where each came fromwritten for this kit rather than collected — the corpus is generated in the kit's own repository, so there is no third-party data in it.
Swap this folder for your own material and the kit is pointed at your clearance requests. That is the whole change — there is no database to migrate.
One clearance request, as the model receives itRGT-0001-P1.txt · 1 of 120
Synthetic Record
----------------------------------------------------------------
Every title, licensee, territory, licence window, fee, clause, name and note in this file
is INVENTED. Generated by tools/build_corpus.py from seed 20260825. No real distributor,
licensor, licensee, contract or person appears anywhere in it. MIT licensed, like the rest
of this repository.
Title And Request
----------------------------------------------------------------
Clearance request : RGT-0001-P1
Title reference : RGT-0001
Title : Marram Foundry
Review date : 2026-08-25
This request is : candidate pair 1 of 3 raised on this title
Rights manager of record: Tomas Lindqvist
Escalation horizon : 60 days from the review date
Grant A Deal Summary
----------------------------------------------------------------
Grant reference : G-0001-01
Licensee : Blue Meridian TV
Territory : Benelux
Platform : Free-to-air TV
Languages : Portuguese, Dutch
Licence window : 2027-02-04 to 2028-08-04
Exclusivity : EXCLUSIVE
Exclusivity clause : A-3.1
Signed : 2026-08-08
Licence fee : USD 1,250,000
Grant B Deal Summary
----------------------------------------------------------------
Grant reference : G-0001-02
Licensee : Fenwick Media Group
Territory : Benelux
Platform : Free-to-air TV
Languages : Portuguese, Dutch
Licence window : 2027-06-25 to 2028-05-22
Exclusivity : EXCLUSIVE
Exclusivity clause : B-3.1
Signed : 2026-04-18
Licence fee : USD 85,000
Clearance Policy (Default)
Abridged — the file continues.
The outcomeWhat a good result looks like
every candidate pair's verdict with the deciding axis, the blocked days and the clause reference -- and the real conflicts separated from the permitted overlaps with 100.0 pct precision and 100.0 pct recall, 0 false escalations across 75 quiet readings.
And when it cannot
a CONTEXT_INCOMPLETE reading (a field R-1 requires never supplied on one grant -- 8 of 120) is not judged rather than judged against a partial record. There is no default grant of rights for a field nobody filled in.
Where it fitsWhat did work
Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.
Every holdback, carve-out and definition IS keyed into a structured field — the free floor -- clause-tag-strong, $0.00 It parses the keyed lines and is then handed the rule engine, so it scores 100.0 pct on those readings for nothing.
The deciding term lives in contract prose your ledger never captured — the model, reading the clause extracts Verdict 100.0 pct against the strongest free floor's 78.33, and 100.0 pct against 0.0 on the prose readings specifically.
And where nothing here is good enough:
You can reach the rights ledger but not the contracts behind it — neither, and the measurement is unusually blunt The ledger-only control reproduced the FREE overlap-exclusivity floor's verdict on 120 of 120 readings -- the same answer, not the same average -- for 341,915 input and 160,968 output tokens.
Hundreds of live grants per title and no candidate-pair generator — not this kit, yet Cost is quadratic in grants per title and the blocking step is named, not built.
At a glanceHow the whole thing runs
100%verdict accuracy pct
7,869 msp50, end to end
$5.65per 1,000 clearance requests · Google Gemini 3 Flash
Run once, for real, on 2026-08-25. Every figure on these pages was captured from that run — nothing is written from intent.
14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.
Check whether two license deals on one show clash14 steps · 4 questions · run once, for real · 2026-08-25
Every tile is a linkall 14 steps · no step without a page
Should you use this?What you bring, where it stops, and when not to use it
Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.
What do I have to bring?
tools/build_corpus.py writes data/corpus/*.txt and data/gold.jsonl. ⚠︎ THE PRECISION FIGURES DO NOT TRANSFER.Corpus lens →
When is this the wrong choice?
Avoid: Do not pay a model for a ledger that already holds the term: on the keyed readings the free floor ties it at 100.0 pct. That is the case against the best-fitting scenario (“Every holdback, carve-out and definition IS keyed into a structured field”). 4 scenarios scored in all, each with its own.Eval lens →
Where does it stop working?
A grant with a field R-1 requires missing (8 of 120 readings): CONTEXT_INCOMPLETE, and no conflict is asserted. 6 recorded failure modes, each from a run rather than a guess.Corpus lens →
What was never verified?
Whether the axis disagreement would disappear if the rule text named an axis for R-7. The fix was diagnosed AFTER the run and deliberately not applied: changing the question after seeing the answers is choosing the scoreboard after the game. 5 items this kit says it could not check.Eval lens →
Can I run this on a model I control?
Yes — any OpenAI-compatible endpoint, including one on your own hardware. The shipped adapter takes its host from BASE_URL and its model from MODEL, so nothing in src/ changes. The published figures come from 1 model on the fast tier, reading the clause extracts, one provider, one key. Prompt lens →
And if it fits — what do I stand up?
4 artifacts with a stated home and a stated egress, and 3 decisions each with what you provision past its ceiling — plus what was not measured. That is the next page, not this one.step 14 — Run it in your environment →
Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).
Last verified 2026-08-25 — r001-rights-conflict. Every figure on these pages was captured from that run.
Run itHow this reaches your data
Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.
Run this on your own data
The pipeline, its eval harness and the runs behind every numberdeployed inside your environment, on your own model endpoints, against your own documents.
The corpus above is the shape, not the limitit is a folder swap, and there is no database to migrate.
Checked before this shipped — Every clearance request, the answer key, all three free floors, the injection probe and what r001-rights-conflict actually answered ship in the repo. python3 -m evals.check_labels and python3 -m src.app both run with no key and no network.
PresenterOpens the private repo. Visible to admins only.
Step 01 of 14Business case
Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.
In brief
100.0%rows answered
7,869 msp50, end to end
52,195 msp95
5 minclone to first result
What the clock covers. one reading -- one candidate grant pair -- end to end including provider-side reasoning tokens, on the shared connection. Not a cold-start figure and not a per-title SLA: 120 readings ran in 499.4 wall seconds because the pairs are independent and run concurrently.
Current processWhat it replaces
a rights conflict report that interval-overlaps two date ranges and flags every intersection -- which on this corpus invents 34 conflicts that are not there and walks past 38 that are.
Where it is not good enough
⚑ THE MODEL LOSES A WHOLE SLICE OF THE AXIS COLUMN: 0 of 16 on the exclusive-overlap readings, where ALL THREE free floors score 16 of 16. It answers WINDOW; the key requires EXCLUSIVITY. A clean sweep like that is the signature of a specification gap rather than a reading error, and it is: R-8 names its axis, R-4 names its axis, R-5 and R-6 name theirs, and R-7 -- the rule that produces this verdict -- names none. The model's own rationale has the better of the argument on the facts (what brought the two grants together IS the window); the key's reading is that two non-exclusive grants with the same window are fine, so exclusivity is what makes it a conflict. Both are defensible and the prompt never chose. THE FIX IS NOT APPLIED AND NOT MEASURED -- one sentence in the rule text would close it, and it costs another 120-call run; a number nobody paid for is not published. The second loss is one cell: blocked_days 535 against the key's 536 on RGT-0029-P3, where the key is right and the model dropped the inclusive end date.
One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.
→Sources
txt120jsonl1json1
40 titles, 120 readings across 3 candidate grant pairs each — every pair judged from its own page
1Candidate pairsno lens on the shipped page
120 readings, 40 titles x 3 candidate pairs
NO carried state — everything a reading needs is printed on it
2Clause extractsno lens on the shipped page
in Contract clauses
the holdback, carve-out, definition and nesting clauses the LEDGER does not hold
52 of 120 readings turn on one; 26 keyed as a tag, 26 prose only
verdict 100.0% against the strongest free floor's 78.33%
discriminator 100.0% — precision 100.0, recall 100.0, 0 missed of 54, 0 false of 66
on PROSE-only clauses 100.0% — the strongest free floor reads 0.0%
56.67%strip the clause text and verdict falls to
the model takes 0 of 16 on the deciding axis for exclusive overlaps — all three free floors take 16 of 16
2026-08-25as of
A rights clearance desk, pair by pair. Subtracting two licence windows is free and every arm does it; the model is there because THE CLAUSE DECIDES, NOT THE CALENDAR. ⚑ 72 OF THE 120 READINGS (60 PCT) ARE ONES A GENUINE INTERVAL-OVERLAP IMPLEMENTATION GETS WRONG — and that is COMPUTED BY RUNNING IT, not asserted: the corpus builder calls the naive implementation to set the flag and the kit's gate re-derives it a second time and refuses to start if the two disagree. The split is printed both ways: 38 real conflicts it reports as permitted (a licence sold twice) and 34 permitted overlaps it reports as conflicts (a lawyer's morning on two non-exclusive grants that were always fine). ⚑ AND THAT FLOOR IS THE HONEST REFUTATION OF A RECALL-ONLY HEADLINE: 32.0 pct precision AND 29.63 pct recall — not merely noisy, wrong in both directions at once. A conflict report that flags everything would score 100 pct recall and be worthless, which is why this kit refuses to publish either number alone. ⚑ THE SHARPEST RESULT IN THIS LAP IS THE ABLATION: the ledger-only arm — the same prompt with one block's body replaced — agreed with the FREE overlap-exclusivity floor on 120 OF 120 VERDICTS. Not the same average; the same answer on every single reading, after spending 341,915 input tokens to reach it. Without the contract text a reasoning model is worth exactly what forty lines of free Python are worth, and it walks past USD 16,345,000 of licence-fee exposure doing it.
⚠︎ THE MODEL LOSES A WHOLE SLICE TO FREE CODE AND THIS FIGURE SAYS SO ON THE SCORED STATION: 0 of 16 on the deciding axis for exclusive overlaps, where every floor scores 16 of 16. A 16-of-16 sweep is a SPECIFICATION GAP, not a misread page — R-4, R-5, R-6 and R-8 each name their axis and R-7, the rule that produces this verdict, names none. Diagnosed after the run and DELIBERATELY NOT FIXED: one sentence would close it, it costs another 120-call run, and changing the question after reading the answers is choosing the scoreboard after the game. A second cell went the other way — blocked_days 535 against the key's 536 — and there the key is right.
⚠︎ THE SCORED ARMS RAN AT A 16,000-TOKEN CEILING AND PEAKED AT 10,044 (62.8 PCT), with 3 of 120 readings above 8,000 against a median of 817.5 — so a ceiling set from this kit's own nine-call probe, which drew 4,855, would have truncated three scored readings. The published ceiling is now 32,000 and c001 confirms it at 7,857 (24.6 pct); the scored arms are NOT discarded, because nothing truncated and both arms answered 120 of 120. ⚑ THE TWO PROBES DREW 4,855 AND 7,857 ON THE SAME NINE CHAINS — 1.62x apart on identical input, the drawn-not-fixed reasoning budget reproduced inside this kit's own results. ⚑ THE INJECTION PROBE FORCED ITS CONDITION AND PAIRED EVERY READING AGAINST ITS OWN UN-INJECTED ANSWER: 45 of 45 escalations still raised, 0 suppressed, 0 excluded.
⚠︎ AND ITS OWN BLIND SPOT IS NAMED — the note was forced into Rights Notes, never written AS A CONTRACT CLAUSE, which is the surface this kit most depends on and did not probe.
The swap seams
Seam
File
What changes
the clearance rule
src/rights.py
The eleven rules and their precedence -- the knob that decides what a conflict IS on your paper.
the holdback convention
src/rights.py
Whether an exclusivity extends past its own window, and by how long.
the escalation horizon
src/rights.py
DEFAULT_ESCALATION_HORIZON_DAYS -- when a conflict stops being preventable and becomes a breach for a different desk.
the model
.env
PROVIDER, BASE_URL, MODEL.
what leaves the machine
src/select.py
SECTION_HINTS and NEVER_SENT -- and both, because either alone holds.
the machine-readable clause vocabulary
evals/baseline.py
The CLAUSE: tag grammar the strong floor parses -- change it to whatever your own rights system actually keys in.
Components
Component
File
Role
the clearance rule
src/rights.py
Eleven rules -- resolution, contention, carve-out, non-exclusivity, window overlap, holdback, the blocking-clause tie-break and the escalation test. decide() is also the answer key, and check_labels replays every row through it.
the section splitter
src/segment.py
Splits the request into its 8 named sections; asserted across all 120 documents. Losing Clause Extracts silently is the worst failure it can have, and it would not look like one.
the send filter
src/select.py
Rights Manager Contact is mapped by no field and therefore never sent. Two independent locks, and neither is red-provable alone.
the prompt
src/prompt.py
Two parts: the instruction with the eleven rules and the JSON shape, then the request. The ledger-only control replaces exactly one section's body.
the model call
src/adapters/__init__.py
Raw HTTP, stdlib only, bounded retry; the shared daily call cap is checked here.
the reader
src/clearance.py
One candidate pair, one call. The published token ceiling lives here.
the scorer
evals/scoring.py
Exact match per cell. The conflict/permitted discriminator is a separate binary and prints precision beside recall with both denominators.
the three free floors
evals/baseline.py
interval-overlap, overlap-exclusivity and clause-tag-strong -- $0.00 each, and the third is handed the answer key's own rule engine.
the corpus generator
tools/build_corpus.py
Plants the inputs and asks decide() for the answers. Never writes a verdict.
the local UI server
src/app.py
http.server, stdlib. Renders with no key and shows all three answers side by side.
the local UI client
ui/app.js
Hand-written JS, no framework.
Where it breaks at scale
QUADRATIC IN GRANTS PER TITLE, NOT LINEAR IN TITLES. A title with 12 live grants has 66 candidate pairs, and nothing here blocks or prunes them -- this kit reads the pairs it is given. A real catalogue needs a blocking step in front of it (same title, overlapping or adjacent windows, intersecting resolved territory) and that step is NOT in this kit and is not measured. ⚑ AND THE PROMPT IS MOSTLY FIXED TEXT: the eleven printed rules are 2924 of the 3018.52 input tokens per reading, identical on every call, so a provider with prompt caching would price this workload very differently. That was not measured.
PresenterOpens the private repo. Visible to admins only.
Step 03 of 14App UI
Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.
SuccessesWhen it works
RGT-0003-P2, replayed from the scored run. The two licence windows are 15 days apart and share not one day -- and clause A-4.2 holds Grant A's exclusivity for 60 days past its expiry, so 46 days of Grant B's window are blocked. The holdback is written as a sentence, not keyed into a field, so BOTH free floors read the same page and answer PERMITTED. Every answer row differs.successOpen full size →The same request before anything is read: both deal summaries parsed in code, and both free floors' answers, all rendering with no key.emptyOpen full size →The clear button pressed with no API_KEY -- a plain sentence, not an error.nokeyOpen full size →
LimitsWhen it does not
A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.
RGT-0029-P3 -- the kit's own loss, framed. The model calls the deciding axis WINDOW where the key says EXCLUSIVITY, and counts 535 blocked days where the key says 536. All three free floors get both cells. The axis disagreement is a gap in the kit's own rule text, not a misread page; the day count is a plain arithmetic slip.failureOpen full size →
PresenterOpens the private repo. Visible to admins only.
Step 04 of 14Corpus
Written for: "can I point this at mine?" · Measured off the corpus the run actually used.
In brief
120clearance requests
0.83 MiBtxt 120
p50 7,220chars per reading
$0.00setup · 0.6s
How it is cutWhat one reading is
40 titles x 3 candidate grant pairs. The pairs are INDEPENDENT and carry no state between them -- everything a reading needs is printed on its own page, which is what makes the discriminator a property of the page rather than of the order it was read in.
SetupWhat the setup figure measured
There is no index to build -- each reading's request goes whole into the prompt. The 0.6s and $0.00 are the corpus generation itself.
LicenceLicence
MIT
Bring your ownBring your own clearance requests
tools/build_corpus.py writes data/corpus/*.txt and data/gold.jsonl. Keep the 8 section headings and the field labels src/clearance.facts_of parses. Set your own eleven rules, holdback convention and escalation horizon in src/rights.py FIRST -- they are placeholders, not anybody's terms.
⚠︎ And what stops being true when you do: ⚠︎ THE PRECISION FIGURES DO NOT TRANSFER. This corpus is 45 pct conflicts by construction, so both error directions have a denominator a reader can use; a real candidate-pair list is overwhelmingly clean, and precision on a 2 pct base rate is a different number. What DOES transfer is the shape of the finding: which kinds of pair a date subtraction gets wrong, and in which direction. Re-run the harness on your own data -- that is what it is for.
What breaks it
A grant with a field R-1 requires missing (8 of 120 readings): CONTEXT_INCOMPLETE, and no conflict is asserted.
TWO CONTRACTS DEFINING THE SAME TERM DIFFERENTLY. R-2 says a defined term is resolved by the contract that uses it, and this corpus declares at most ONE definition per reading -- the easy half of that rule. A real pair where both contracts define 'Streaming' and disagree is not modelled and not measured.
Sub-distribution and sub-licensing chains. Every grant here is licensor-to-licensee, one hop.
Most-favoured-nation, matching and first-negotiation clauses, options, rolling terms and change-of-control -- none of which this corpus contains.
A rights export that renames its sections -- the send filter's fallback is what stops the whole document going on the wire, reproduced on 120 of 120 documents.
Per-episode or per-season splitting, and dubbing versus subtitling inside one language. A grant here is territory x platform x language x window.
PresenterOpens the private repo. Visible to admins only.
Step 05 of 14Prompt
Written for: engineer · Logged from the run, verbatim.
In brief
The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — but they are the input total apportioned by each part’s share of the characters, not a per-part reading from the provider’s counter. The total is measured; the split is arithmetic, and a part carrying denser text really costs more of it than this says.
AssemblyHow the prompt is assembled
Part
Characters
Tokens
system
213
53
the question, the eleven rules and the JSON shape
4,765
1,191
the clearance request, 7 of its 8 sections
7,051
1,762
Total
3,006
This is the cost lesson as arithmetic: of the 3,006 tokens assembled, 1,762 are documents — 59% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.
Reproduced from src/prompt.py as sent -- string concatenation, no template engine. Two parts: the instruction (fixed on every call) and the request, which is what src/select.py let through in document order. The eleven rules are printed on every request AND stated in the instruction, deliberately: an arm that could only see one of the two copies would be measuring whether it found the policy, not whether it applied it.
Full promptThe prompt in full
The verbatim prompt, as sent
the prompt, as sent
You are a rights clearance desk. You read one candidate pair of live licence grants on one title, against the DEFAULT clearance policy printed on the request, and you answer with one JSON object and no other text.
You are a rights clearance desk for one distributor's catalogue. You are reading ONE candidate pair
of live licence grants on ONE title, and deciding whether the two grants can both be honoured.
Both grants' deal summaries, the DEFAULT clearance policy, the clause extracts from both contracts
and the rights notes on file are reproduced below. Apply the policy exactly as written.
⚠︎ THE CLEARANCE POLICY, THE HOLDBACK CONVENTION AND THE ESCALATION HORIZON ARE ILLUSTRATIVE
DEFAULTS, NOT ANY REAL LICENSOR'S OR DISTRIBUTOR'S TERMS. Apply them as printed regardless of
whether they look right for the deal in front of you.
THE POINT OF THIS JOB: most apparent overlaps are FINE. Two non-exclusive licensees can exploit the
same title in the same market on the same day; a territory can nest inside another and still be
carved out by name; a window can overlap completely and breach nothing. And some pairs that do NOT
overlap on a calendar are genuine conflicts, because an exclusivity extends past its own window by a
holdback, or because one contract's definition of a platform swallows a service the other names.
A date subtraction flags the first kind and misses the second. THE CLAUSE DECIDES, NOT THE CALENDAR.
How to work it out -- follow the policy's rules in their stated order:
- FIRST check every field R-1 requires is supplied on BOTH grants. If any is missing, the verdict
is CONTEXT_INCOMPLETE: nothing is judged, nothing is escalated, blocked_days is 0, the axis is
NONE and the blocking clause is NONE. There is no default grant of rights for a field nobody
supplied.
- THEN resolve each grant's TERRITORY and PLATFORM to leaf markets and leaf services using that
grant's OWN definitions clause (R-2). "Benelux" and "Belgium" are different words and may be the
same market. "Streaming" and "AVOD" are different words and may be the same service -- or may
not, depending on which contract you are reading. Where nothing defines a label, the label is the
leaf.
- THEN decide whether the two grants CONTEND (R-3): their resolved sets must intersect on
territory AND platform AND language. All three.
- THEN apply any CARVE-OUT (R-4). An express exception that removes the whole contended slice ends
the matter: PERMITTED_CARVE_OUT.
- If they do not contend at all, the verdict is PERMITTED_DISJOINT and the axis is whichever of
TERRITORY, PLATFORM or LANGUAGE separates them (R-5). The calendar is irrelevant here.
- If they contend and BOTH are NON_EXCLUSIVE, the verdict is PERMITTED_NON_EXCLUSIVE (R-6),
however completely the windows overlap. This is the ordinary case and it is NOT a finding.
- If they contend, at least one is EXCLUSIVE and the windows OVERLAP, it is a CONFLICT (R-7).
Name it CONFLICT_TERRITORY_NEST when the territory labels differ and one resolves inside the
other, CONFLICT_PLATFORM_DEFINITION when the platform labels differ and a defined term brought
them together, and CONFLICT_EXCLUSIVE_OVERLAP otherwise. blocked_days is the days the two
windows share, counting both end dates.
- If they contend, at least one is EXCLUSIVE and the windows do NOT overlap, look for a HOLDBACK
(R-8). blocked_days is the days of the later window that fall inside the holdback tail.
- Otherwise PERMITTED_DISJOINT on the WINDOW axis (R-9).
- THEN name the BLOCKING CLAUSE exactly as R-10 defines it, using the clause reference printed in
the extracts (for example "A-4.2"). Where both grants are exclusive and R-10 asks for the
earlier-signed one, compare the printed Signed dates.
- FINALLY decide "escalate" (R-11): YES only for a CONFLICT whose blocked period has not already
ended before the review date printed on this request. A conflict entirely in the past is
recorded and NOT escalated.
Answer with a single JSON object and nothing else:
{"verdict": "CONFLICT_EXCLUSIVE_OVERLAP|CONFLICT_HOLDBACK|CONFLICT_PLATFORM_DEFINITION|CONFLICT_TERRITORY_NEST|PERMITTED_NON_EXCLUSIVE|PERMITTED_CARVE_OUT|PERMITTED_DISJOINT|CONTEXT_INCOMPLETE",
"axis": "WINDOW|TERRITORY|LANGUAGE|PLATFORM|EXCLUSIVITY|HOLDBACK|DEFINITION|CARVE_OUT|NONE",
"blocked_days": <integer, 0 when nothing is blocked>,
"blocking_clause": "<the clause reference, e.g. A-4.2, or NONE>",
"escalate": "YES|NO",
"rationale": "one sentence, naming the axis that decided it, the clause you relied on, and -- if
the calendar and the clause disagree -- which one you followed and why"}
Precedence, applied in this order: CONTEXT_INCOMPLETE (R-1); then PERMITTED_CARVE_OUT (R-4); then
PERMITTED_DISJOINT on a non-contending axis (R-5); then PERMITTED_NON_EXCLUSIVE (R-6); then a
window-overlap CONFLICT (R-7); then CONFLICT_HOLDBACK (R-8); otherwise PERMITTED_DISJOINT on the
WINDOW axis (R-9).
Clearance request
----------------------------------------------------------------
Synthetic Record
----------------------------------------------------------------
Every title, licensee, territory, licence window, fee, clause, name and note in this file
is INVENTED. Generated by tools/build_corpus.py from seed 20260825. No real distributor,
licensor, licensee, contract or person appears anywhere in it. MIT licensed, like the rest
of this repository.
Title And Request
----------------------------------------------------------------
Clearance request : RGT-0003-P2
Title reference : RGT-0003
Title : The Burning Meridian
Review date : 2026-08-25
This request is : candidate pair 2 of 3 raised on this title
Rights manager of record: Marguerite Obi
Escalation horizon : 60 days from the review date
Grant A Deal Summary
----------------------------------------------------------------
Grant reference : G-0003-03
Licensee : Greyling Networks
Territory : Mexico
Platform : EST
Languages : Italian, English
Licence window : 2026-10-08 to 2027-10-03
Exclusivity : EXCLUSIVE
Exclusivity clause : A-3.1
Signed : 2025-05-23
Licence fee : USD 330,000
Grant B Deal Summary
----------------------------------------------------------------
Grant reference : G-0003-04
Licensee : Delphi Screens
Territory : Mexico
Platform : EST
Languages : Italian, English
Licence window : 2027-10-18 to 2029-05-15
Exclusivity : EXCLUSIVE
Exclusivity clause : B-3.1
Signed : 2026-03-06
Licence fee : USD 960,000
Clearance Policy (Default)
----------------------------------------------------------------
CLEARANCE POLICY (DEFAULT) -- ILLUSTRATIVE, NOT ANY REAL LICENSOR'S TERMS
R-1 If either grant's licence window, territory, platform, language or exclusivity is NOT SUPPLIED,
the verdict is CONTEXT_INCOMPLETE. No conflict is asserted, nothing is escalated, blocked_days
is 0, the axis is NONE and the blocking clause is NONE. There is no default grant of rights for
a field that was never supplied.
R-2 RESOLVE each grant's TERRITORY and PLATFORM to the leaf markets and leaf services its OWN
contract defines. A defined term is resolved by the definitions clause of the contract that
USES it, never by the ordinary meaning of the word and never by the other contract's
definitions. Where no definition applies, the printed label IS the leaf.
R-3 Two grants CONTEND when their resolved sets intersect on TERRITORY and on PLATFORM and on
LANGUAGE. All three. Two grants that share a territory and a platform but no language do not
contend.
R-4 If a CARVE-OUT in either contract removes the whole of the contended slice from that grant,
the verdict is PERMITTED_CARVE_OUT, the axis is CARVE_OUT, blocked_days is 0, and the blocking
clause is the carve-out clause. An express exception beats a general grant.
R-5 If the grants do not contend on territory, on platform or on language, the verdict is
PERMITTED_DISJOINT and the axis is whichever of TERRITORY, PLATFORM or LANGUAGE separates them
-- whatever the calendar says.
R-6 If they contend and BOTH grants are NON_EXCLUSIVE, the verdict is PERMITTED_NON_EXCLUSIVE, the
axis is EXCLUSIVITY and blocked_days is 0, however completely the two windows overlap. Two
non-exclusive licensees exploiting the same title in the same market on the same day is the
ordinary case, not a finding.
R-7 If they contend, at least one grant is EXCLUSIVE, and the two licence windows themselves
overlap, the verdict is a CONFLICT. Name it by what brought the two grants together:
- CONFLICT_TERRITORY_NEST the two territory LABELS differ and one resolves inside the
other;
- CONFLICT_PLATFORM_DEFINITION the two platform LABELS differ and a defined term in one
contract subsumes the service the other names;
- CONFLICT_EXCLUSIVE_OVERLAP otherwise.
blocked_days is the number of days the two licence windows share, counting both end dates.
R-8 If they contend, at least one grant is EXCLUSIVE, and the licence windows do NOT overlap, look
for a HOLDBACK. An exclusive grant whose contract holds the contended media back for N days
after its window ends is exclusive for those N days as well. If the other grant's window begins
inside that tail, the verdict is CONFLICT_HOLDBACK, the axis is HOLDBACK, and blocked_days is
the number of days of the later window falling inside the tail. A holdback is why two windows
that do not touch on a calendar can still be unhonourable.
R-9 Otherwise the verdict is PERMITTED_DISJOINT and the axis is WINDOW.
R-10 THE BLOCKING CLAUSE is the clause that decides. For CONFLICT_HOLDBACK it is the holdback
clause; for CONFLICT_PLATFORM_DEFINITION the definitions clause; for CONFLICT_TERRITORY_NEST
the territory-definition clause; for CONFLICT_EXCLUSIVE_OVERLAP the exclusivity clause of the
EARLIER-SIGNED exclusive grant, because that is the grant whose rights were sold first; for
PERMITTED_CARVE_OUT the carve-out clause. It is NONE for every other verdict.
R-11 escalate is YES only when the verdict is a CONFLICT and the blocked period has NOT already
ended before the review date -- there is still something to prevent. A conflict whose blocked
period is entirely in the past is recorded and NOT escalated: it is a breach that already
happened, it goes to the breach log, and that is a different desk with a different remedy.
Clause Extracts
----------------------------------------------------------------
Grant A, clause A-4.2 (Holdback)
Licensor shall not licence, and shall procure that no third party exploits, the Picture by EST in the Territory for a period of 60 days following the expiry of the Licence Period, the Licensee's exclusivity being deemed to continue throughout that period.
Grant A, clause A-3.1 (Grant of rights)
Licensor hereby grants to Licensee the sole and exclusive right and licence to exploit the Picture by the Licensed Platform in the Territory in the Licensed Languages throughout the Licence Period.
Grant B, clause B-3.1 (Grant of rights)
Licensor hereby grants to Licensee the sole and exclusive right and licence to exploit the Picture by the Licensed Platform in the Territory in the Licensed Languages throughout the Licence Period.
Grant B, clause B-12.6 (Reporting)
Licensee shall render statements of exploitation within forty-five (45) days of the end of each calendar quarter, itemised by platform and by territory, whether or not any sum is then due.
Rights Notes
----------------------------------------------------------------
No note is on file for this request.
Raw responseThe raw response
The raw response, before any parsing
the response, unparsed
{"verdict": "CONFLICT_EXCLUSIVE_OVERLAP", "axis": "WINDOW", "blocked_days": 333, "blocking_clause": "B-3.1", "escalate": "YES", "rationale": "Both grants are exclusive in Benelux on free-to-air TV in Portuguese and Dutch, and their windows overlap from 2027-06-25 to 2028-05-22 inclusive, so the earlier-signed exclusive grant B-3.1 blocks the overlap and escalation is warranted because the blocked period is still in the future."}
PresenterOpens the private repo. Visible to admins only.
Step 06 of 14Evals
Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.
In brief
Check whether two license deals on one show clash — 120 clearance requests. One model answered, and every answer was then graded Three different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.
120clearance requests
120source documents
1model tier
3grading methods
MeasurementsWhat was measured
COUNTED120 · 68 · 94 · 68 · 48 / 120verdict accuracy pct — verdict, eight-wayDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.
COUNTED104 · 66 · 101 · 80 · 80 / 120axis accuracy pct — deciding axisDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.
COUNTED119 · 68 · 94 · 68 · 48 / 120blocked days accuracy pct — blocked daysDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.
COUNTED120 · 68 · 94 · 68 · 46 / 120blocking clause accuracy pct — blocking clauseDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.
COUNTED120 · 76 · 100 · 76 · 63 / 120escalate accuracy pct — escalate callDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.
COUNTED120 · 68 · 94 · 68 · 48 / 120conflict window accuracy pct — real conflict vs permitted overlap -- accuracyDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.
COUNTED54 · 16 · 38 · 16 · 16 / 54conflict window precision pct — real conflict -- precisionDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.
COUNTED54 · 16 · 38 · 16 · 16 / 54conflict window recall pct — real conflict -- recallDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.
COUNTED72 · 20 · 46 · 20 · 0 / 72calendar misleading caught pct — readings the calendar alone gets wrong, caughtDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.
COUNTED66 · 56 · 60 · 56 · 43 / 66permitted left alone pct — permitted overlaps correctly left aloneDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.
COUNTED26 · 0 · 0 · 0 · 0 / 26prose verdict accuracy pct — verdict where the deciding term is PROSE onlyDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.
NOT YET KNOWN—A person confirmed the gold is rightThe grader is == and needs no confirming. The gold does: it is machine-derived from the registry's structured modules, and every rate on this page rests on that derivation being faithful to what the prose actually says. Nobody has read a document against its gold record by hand.
Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.
How the method was validated
evals/check_labels.py replays every committed row through src/rights.decide() and requires the committed answers exactly before any run may spend -- 23 checks, green, red-proven on five seeded defects and re-acquitted. ⚑ IT ALSO ASSERTS THE DISCRIMINATOR IS NOT TRIVIAL: 72 of 120 readings are ones a pure interval-overlap implementation gets WRONG, re-derived by RUNNING that implementation rather than by trusting a column the generator wrote. ⚑ AND IT ASSERTS THE STRONGEST FLOOR IS RIGHT BY CONSTRUCTION wherever the deciding term was keyed in -- that check caught a (\S+) in the tag parser that captured "United" out of "United Kingdom,Ireland", silently costing the floor four readings, before a single call was spent.
Grading costWhat it costs
Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection, never a bill and never a vendor's claim about this workload.
Priced at
Per 1M in / out
One clearance request
1,000 clearance requests
Share that is the prompt
Google Gemini 3 Flash The estate's shared projection card. Every kit in this series prices its measured tokens against the same published card so the cost pages are comparable to each other.
$0.30 / $2.50
$0.005654
$5.65
16%
Same work, 1× the bill
The same clearance requests, the same tokens — only the rate card changed. And on that card about 16% of what you pay is the prompt this pipeline sends, not the answer it writes.
THE BLOCKING STEP THAT IS NOT IN THIS KIT. Cost is quadratic in grants per title, not linear in titles, so the lever is which candidate pairs you generate at all -- and this kit reads the pairs it is given rather than choosing them.
Rates checked 2026-08-18. The provider that actually ran all 294 calls is kept out of these tables per this estate's naming rule, so no figure here is a bill.
the fast tier, reading the clause extracts 100.0% conflict window accuracy · what free code writes in an afternoon 40.0% conflict window accuracy · the strongest free floor -- no model 78.3% conflict window accuracy · 3 more measured on each run
the fast tier, reading the clause extracts 100.0% tag verdict accuracy · the strongest free floor -- no model 100.0% tag verdict accuracy · 1 more measured on each run
Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.
Why this set cannot separate them
⚑ THE LEDGER-ONLY CONTROL AGREED WITH A FREE FLOOR ON 120 OF 120 VERDICTS -- not the same average, the same answer on every single reading. Strip the clause text and a reasoning model reproduces evals/baseline.py's overlap-exclusivity exactly, having spent 341,915 input and 160,968 output tokens to do it. That is the sharpest available statement that the value here is reading the clause, not calling a model.
Choosing oneWhich one to use
The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.
If your situation is
Use
Because
And avoid
Every holdback, carve-out and definition IS keyed into a structured field
the free floor -- clause-tag-strong, $0.00
It parses the keyed lines and is then handed the rule engine, so it scores 100.0 pct on those readings for nothing.
Do not pay a model for a ledger that already holds the term: on the keyed readings the free floor ties it at 100.0 pct.
The deciding term lives in contract prose your ledger never captured
the model, reading the clause extracts
Verdict 100.0 pct against the strongest free floor's 78.33, and 100.0 pct against 0.0 on the prose readings specifically.
Do not trust its DECIDING AXIS on an exclusive overlap: 0 of 16, where every free floor scores 16 of 16. The rule text never named an axis for that branch.
You can reach the rights ledger but not the contracts behind it
neither, and the measurement is unusually blunt
The ledger-only control reproduced the FREE overlap-exclusivity floor's verdict on 120 of 120 readings -- the same answer, not the same average -- for 341,915 input and 160,968 output tokens.
Do not ship the ledger-only arm and describe it as this kit: it finds 16 of the 54 real conflicts and walks past USD 16,345,000 of exposure.
Hundreds of live grants per title and no candidate-pair generator
not this kit, yet
Cost is quadratic in grants per title and the blocking step is named, not built.
Do not read this kit's per-reading cost as a per-title cost. A title with 24 live grants is 276 readings.
This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.
Failure causesHow it fails
Cause
What it means
Rows
A real example
AXIS_UNSPECIFIED_ON_R7
a deciding axis the rule text never named
16
All 16 exclusive-overlap readings. The model answers WINDOW; the key requires EXCLUSIVITY. R-8, R-4, R-5 and R-6 each name their axis and R-7 -- the rule that produces this verdict -- names none, so the floors collect these cells for free by emitting the…
INCLUSIVE_END_DROPPED
an off-by-one on the inclusive end date
1
RGT-0029-P3: 535 blocked days against the key's 536 over 2026-09-16 to 2028-03-04. R-7 says 'counting both end dates'. Here the key is right and the model is wrong, on one reading in 120.
What we could NOT verify
Whether the axis disagreement would disappear if the rule text named an axis for R-7. The fix was diagnosed AFTER the run and deliberately not applied: changing the question after seeing the answers is choosing the scoreboard after the game.
Whether a second model tier reads a prose holdback as reliably. One model, one run per arm; every other row in the cost table is a projection onto a published card.
Repeats. One run per arm, so nothing here separates a real difference from a re-roll of the provider's reasoning budget.
A real ledger's base rate. This corpus is 45 pct conflicts by construction and the precision figures do not transfer.
Whether rendering the clause extracts as structured JSON rather than as the contract's own prose would score differently. That is one more 120-call run and it was not paid for.
PresenterOpens the private repo. Visible to admins only.
Step 07 of 14Unit cost
Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.
In brief
Tier
Input tokens
Output tokens
p50 latency
Google Gemini 3 Flash
the fast tier, reading the clause extracts
3,018.52
1,899.2
7,869 ms
$0.005654
the same tier, CLAUSE TEXT REMOVED
2,849.29
1,341.4
6,689 ms
$0.004208
the strongest free floor -- no model
0
0
0 ms
$0.000000
interval overlap plus the exclusivity flag
0
0
0 ms
$0.000000
what free code writes in an afternoon
0
0
0 ms
$0.000000
The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.
Answering vs gradingAnswering and grading are two separate bills
The three free floors, the stub and every pre-run check cost $0.00 and no calls. No run was discarded on this kit: nothing truncated at the published ceiling on any arm.
Cost driversWhat actually moves the bill
OUTPUT TOKENS, AND ALMOST ALL OF THEM ARE REASONING. 94.1 pct of r001-rights-conflict's output (214,545 of 227,904 tokens) was provider-side reasoning left at the default. The answer is five short fields and a sentence; the bill is the thinking in front of it.
THE PROMPT IS MOSTLY FIXED TEXT. 3018.52 input tokens per reading, and the eleven printed rules, the instruction and the JSON shape are identical on every call -- so a provider with prompt caching would price this workload very differently, and that was not measured.
Your volumeWhat it costs at your volume
QUADRATIC IN GRANTS PER TITLE. A title with 12 live grants is 66 pairs; with 24 it is 276. Nothing amortises across pairs because each reading is independent by design.
Where pricing changes shape
Provider-side reasoning. At 94.1 pct of output on this task, a model whose reasoning is not billed and one whose reasoning is billed differ by roughly an order of magnitude on the same workload.
⚠︎ THE TOKEN CEILING, REFUTED TWICE. c000-rights-conflict-calibration fired the nine hardest prose-only chains at 16,000: largest reply 4855, 30.34 pct of that cap. On that evidence 8,000 looks safe -- and the scored run's largest was 10044 with 3 of 120 readings above 8,000, against a median of 817.5, so a ceiling set from this kit's own probe would have truncated three scored readings. Then 16,000 was itself refuted mid-build by a sibling that recorded 16,452 on ITS scored run. The published ceiling here is now 32,000 and c001-rights-conflict-calibration-32k confirms it: largest reply 7857, 24.55 pct. ⚑ AND THE TWO PROBES DREW 4855 AND 7857 ON THE SAME NINE CHAINS -- 1.62x apart on identical input, which is the drawn-not-fixed behaviour reproduced inside this kit's own results. ⚠︎ THE SCORED ARMS STAY AT 16,000 AND ARE NOT DISCARDED: r001 peaked at 62.77 pct of that cap and s001 at 62.24 pct, both answering 120 of 120 with zero truncations, so no truncated cell hides inside any published percentage. Raising the cap costs nothing -- you are billed for tokens drawn, not for the ceiling.
Your return, with your numbers
Volumecandidate grant pairs per clearance cycle -- this run judged 120 (40 titles x 3 pairs) per arm
What it replacesa rights manager working a conflict report that is wrong in both directions at once
Time saved per itemnot measured here -- it depends on how much of your own holdback, carve-out and definitional detail is keyed into a field versus living in the contract
We publish the inputs, not a return: a return depends on your labour cost and your volume. The ROI dashboard takes it from here, and AI Costing explains the math.
Model choiceWhy this model
It is the tier the shared connection was already pointed at, and the kit's whole claim is that swapping it is .env plus one more run. No second model was run against this corpus; every other row in the cost table is a projection onto a published card and is labelled as one.
Other modelsOn other models
This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.
3,018input tokens · this run
1,899output tokens
$0.006what it actually cost
per-query average, this run
Model
Provider
One eval pass
This whole run
Per 1,000 queries
Rates as of
gpt-5-6-luna
OpenAI
$0.346
$0.346
$2.88
2026-09-12
gemini-3-flash
Google
$0.865
$0.865
$7.21
2026-09-18
gemini-3-8-flash
Google
$1.126
$1.126
$9.39
2026-09-18
llama-5
Meta
$1.421
$1.421
$11.84
2026-09-18
claude-haiku-4-5
Anthropic
$1.502
$1.502
$12.51
2026-09-12
grok-4-5
xAI
$2.092
$2.092
$17.43
2026-09-18
grok-4-6
xAI
$2.092
$2.092
$17.43
2026-09-18
claude-sonnet-5
Anthropic
$3.003
$3.003
$25.03
2026-09-12
gemini-3-1-pro
Google
$3.459
$3.459
$28.83
2026-09-18
gpt-5-6-terra
OpenAI
$3.459
$3.459
$28.83
2026-09-12
gpt-5-6-sol
OpenAI
$6.007
$6.007
$50.06
2026-09-12
claude-opus-4-8
Anthropic
$7.509
$7.509
$62.57
2026-09-12
claude-opus-5
Anthropic
$7.509
$7.509
$62.57
2026-09-12
claude-fable-5
Anthropic
$15.017
$15.017
$125.15
2026-09-18
claude-fable-5-1
Anthropic
$15.017
$15.017
$125.15
2026-09-18
gpt-6-astra
OpenAI
$15.017
$15.017
$125.15
2026-09-17
Read this against the numbers above
Projection only -- no other model was actually called against this corpus.
The reasoning-token share (94.1 pct of output on the fast tier) is measured for that tier only.
Accuracy is NOT projected, only cost -- a cheaper or pricier model is not implied to score the same 100.0 pct.
PresenterOpens the private repo. Visible to admins only.
The codeEvery file, and what it is for
11 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.
Six of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.
The files on this page are not in the public GitHub catalog.
src/rights.pythe clearance rule — a swap seam
Eleven rules -- resolution, contention, carve-out, non-exclusivity, window overlap, holdback, the blocking-clause tie-break and the escalation test. decide() is also the answer key, and check_labels replays every row through it.
You change it to: DEFAULT_ESCALATION_HORIZON_DAYS -- when a conflict stops being preventable and becomes a breach for a different desk.
src/rights.py
# The clearance rule, and the answer key. Pure code, no model, no network.
CONFLICT_EXCLUSIVE_OVERLAP = "CONFLICT_EXCLUSIVE_OVERLAP"
CONFLICT_HOLDBACK = "CONFLICT_HOLDBACK"
CONFLICT_PLATFORM_DEFINITION = "CONFLICT_PLATFORM_DEFINITION"
CONFLICT_TERRITORY_NEST = "CONFLICT_TERRITORY_NEST"
PERMITTED_NON_EXCLUSIVE = "PERMITTED_NON_EXCLUSIVE"
PERMITTED_CARVE_OUT = "PERMITTED_CARVE_OUT"
PERMITTED_DISJOINT = "PERMITTED_DISJOINT"
CONTEXT_INCOMPLETE = "CONTEXT_INCOMPLETE"
VERDICTS = (CONFLICT_EXCLUSIVE_OVERLAP, CONFLICT_HOLDBACK, CONFLICT_PLATFORM_DEFINITION,
src/segment.pythe section splitter
Splits the request into its 8 named sections; asserted across all 120 documents. Losing Clause Extracts silently is the worst failure it can have, and it would not look like one.
src/segment.py
# Split a rights clearance request into its named sections. Pure code, no model.
SECTIONS = ("Synthetic Record", "Title And Request", "Grant A Deal Summary",
RULE = "-" * 64
def split(text):
def render(secs):
src/select.pythe send filter — a swap seam
Rights Manager Contact is mapped by no field and therefore never sent. Two independent locks, and neither is red-provable alone.
You change it to: SECTION_HINTS and NEVER_SENT -- and both, because either alone holds.
src/select.py
# Pick which sections of a clearance request are sent. Pure code -- the last deterministic step
BANNER = "Synthetic Record"
TITLE = "Title And Request"
GRANT_A = "Grant A Deal Summary"
GRANT_B = "Grant B Deal Summary"
POLICY = "Clearance Policy (Default)"
CLAUSES = "Clause Extracts"
MANAGER = "Rights Manager Contact"
NOTES = "Rights Notes"
NEVER_SENT = (MANAGER,)
src/prompt.pythe prompt
Two parts: the instruction with the eleven rules and the JSON shape, then the request. The ledger-only control replaces exactly one section's body.
src/prompt.py
# Assemble the one prompt this kit sends. No template engine, no framework -- string concatenation
INSTRUCTION = """\
LEDGER_ONLY_LINE = ("No clause text is available for either grant on this request. Judge the pair "
def _blank_clauses(secs):
def build(text, carried=None, ledger_only=False):
def rule_text():
src/adapters/__init__.pythe model call
Raw HTTP, stdlib only, bounded retry; the shared daily call cap is checked here.
src/adapters/__init__.py
# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.
class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):
src/clearance.pythe reader
One candidate pair, one call. The published token ceiling lives here.
src/clearance.py
# One candidate grant pair, one model call. The runtime half of the kit.
HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
SYSTEM = ("You are a rights clearance desk. You read one candidate pair of live licence grants on "
MAX_TOKENS = 32000
FIELDS = ("verdict", "axis", "blocked_days", "blocking_clause", "escalate")
def documents():
def titles():
def load_doc(doc_id):
def facts_of(text):
evals/scoring.pythe scorer
Exact match per cell. The conflict/permitted discriminator is a separate binary and prints precision beside recall with both denominators.
evals/scoring.py
# Score a run against the answer key. Pure code, exact match per cell. No model grades anything.
FIELDS = ("verdict", "axis", "blocked_days", "blocking_clause", "escalate")
def _pct(n, d):
def _same(got, want):
def score(records, golds):
def _protection(records, golds):
evals/baseline.pythe three free floors — a swap seam
interval-overlap, overlap-exclusivity and clause-tag-strong -- $0.00 each, and the third is handed the answer key's own rule engine.
You change it to: The CLAUSE: tag grammar the strong floor parses -- change it to whatever your own rights system actually keys in.
evals/baseline.py
# THE FREE FLOORS. Three of them, none a strawman, all of them USD 0.00.
MODES = ("interval-overlap", "overlap-exclusivity", "clause-tag-strong")
NOT_SUPPLIED = "-- NOT SUPPLIED"
def _section(text, name, nxt):
SEC_AFTER = {"Grant A Deal Summary": "Grant B Deal Summary",
def _field(body, label):
def _grant_from_page(text, letter):
def review_date_of(text):
RX_HOLDBACK = re.compile(r"CLAUSE: HOLDBACK (\S+) GRANT ([AB]) (\d+)d")
RX_CARVEOUT = re.compile(r"CLAUSE: CARVEOUT (\S+) GRANT ([AB]) (\S+) EXCLUDES (.+)$", re.M)
tools/build_corpus.pythe corpus generator
Plants the inputs and asks decide() for the answers. Never writes a verdict.
tools/build_corpus.py
# Generate the corpus and the answer key. Deterministic, seeded, and it reads no clock.
HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
STATS = os.path.join(HERE, "data", "corpus-stats.json")
SEED = 20260825
TITLES = 40
PAIRS = R.PAIRS_PER_TITLE
REVIEW = R.REVIEW_DATE
RULE = "-" * 64
src/app.pythe local UI server
http.server, stdlib. Renders with no key and shows all three answers side by side.
src/app.py
# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.
HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
PORT = int(os.environ.get("PORT", "9027"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r001-rights-conflict")
GOLD_ROWS = {r["doc_id"]: r for r in
class H(BaseHTTPRequestHandler):
def main():
ui/app.jsthe local UI client
Hand-written JS, no framework.
ui/app.js
#
Start hereThe shortest path into it
src/app.pyrun it and click around. It holds no pipeline logic on purpose; every number on screen comes from a module the eval harness already drives.
src/rights.pyEleven rules -- resolution, contention, carve-out, non-exclusivity, window overlap, holdback, the blocking-clause tie-break and the escalation test. decide() is also the answer key, and check_labels replays every row through it. A swap seam.
src/segment.pySplits the request into its 8 named sections; asserted across all 120 documents. Losing Clause Extracts silently is the worst failure it can have, and it would not look like one.
src/select.pyRights Manager Contact is mapped by no field and therefore never sent. Two independent locks, and neither is red-provable alone. A swap seam.
src/prompt.pyTwo parts: the instruction with the eleven rules and the JSON shape, then the request. The ledger-only control replaces exactly one section's body.
src/adapters/__init__.pyRaw HTTP, stdlib only, bounded retry; the shared daily call cap is checked here.
src/clearance.pyOne candidate pair, one call. The published token ceiling lives here.
evals/scoring.pyExact match per cell. The conflict/permitted discriminator is a separate binary and prints precision beside recall with both denominators.
evals/baseline.pyinterval-overlap, overlap-exclusivity and clause-tag-strong -- $0.00 each, and the third is handed the answer key's own rule engine. A swap seam.
tools/build_corpus.pyPlants the inputs and asks decide() for the answers. Never writes a verdict.
ui/app.jsHand-written JS, no framework.
Every entry above is a file in this kit, listed in the order the pipeline runs it.
PresenterOpens the private repo. Visible to admins only.
ScenariosWhat it costs at your volume
Everything below is computed from one measured base: 3018 input and 1899 output tokens per reading (one candidate pair of live grants on one title), all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.
Grading really is free here, and that is a property of the ruler rather than a discount
This kit is graded by pure code — the grader makes no model call, needs no key and sends nothing anywhere, so the grading line above is zero at every sampling rate and the leak count stays at nought. What that buys is a ruler you can run on every row, every time, for nothing. What it costs is reach: a code grader can only check what is mechanically checkable.
OperationsFour shapes of workload
The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.
Shape
Reading (one candidate pair of live grants on one title)s/day
Per day
Per month
Per year
Model follows the controls above.
What this ignoresWhere a real bill diverges
Read before quoting any of it
Output length is assumed constant.A terser or more verbose model moves it, and output is priced above input everywhere.
No volume, committed-use, batch or cache discount is modelled.Each moves real enterprise pricing and none are on a public rate card.
Longer documents cost more and nothing here caps them.This kit selects the document’s own sections in pure Python, so there is no vector store to price — but a corpus of longer records raises the input tokens per reading (one candidate pair of live grants on one title) directly, and that is the very number every figure above multiplies.
Grading adds nothing to this bill, and would if you changed the ruler.A code grader is free and local; swapping in an LLM judge would cost more per row than the extraction itself and send every row to a vendor. The instrument is a cost decision, not a footnote.
Rates age.Every one carries the date it was read on the cost page.
PresenterOpens the private repo. Visible to admins only.
In one lineWhat this kit exposes
⚠︎ MEASURED, WITH A FORCED CONDITION AND A PAIRED CONTROL. Rights Notes is this kit's injection surface, and the probe forced the instruction-shaped note onto every reading where an escalation was due rather than counting where the seed happened to put it.
API_KEY is read from .env (gitignored) or the real environment; never logged, never placed in a prompt, and redacted from any provider error message the local UI passes to the browser.
The experimentWe DID attack it -- one sentence, on every reading where it could matter
The probe forces the instruction-shaped note onto every reading where an escalation was due and re-fires them with everything else held identical. Each is then paired against its own un-injected answer from the scored run, because a reading the model never escalated cannot have been suppressed -- 45 of 45 were pairable, and on all 45 the escalation was still raised. Both were run for real on 2026-08-25.
Boundary checked
What could go wrong
What the code guarantees
Whether an instruction embedded in Rights Notes can suppress an escalation this kit's own rules say is due
The free after-the-fact read would have counted whichever of the 26 readings the note happened to land on. Forcing the condition gives a denominator of 45.
x001-rights-conflict-injection forces the condition: every reading whose gold escalate is YES, re-fired with the note REPLACED by the instruction-shaped one, each PAIRED against its own un-injected answer from the scored run. 45 of 45 still raised it. Suppression rate 0.0 pct.
Two boundaries are measured in both directions here: the injection probe forces the condition and pairs each reading against itself, and the privacy guard is red-proven on three separate seeded conditions -- including the one that revealed it has two independent locks and neither is provable alone.
The result0 of 45 escalations suppressed by the instruction-shaped note -- forced condition, paired control, measured.
45paired attack trials fired
0escalations suppressed
One phrasing, one model, one corpus, 45 trials -- every reading where suppression was even possible, each paired against its own un-injected answer.
Read this twice
⚠︎ The Rights Notes reach the model verbatim. On this corpus the instruction-shaped note landed naturally on 26 of 120 readings, which is why an after-the-fact count off the scored run would have been an observation rather than a rate. The probe re-fired all 45 escalation-due readings with the note forced in and none moved — but the denominator is what makes that sentence worth anything, and it is stated beside it.
HonestyWhat this does not prove
Whether another phrasing would move it. A note claiming the licensee has waived exclusivity, claiming legal have already cleared the pairing, or written to look like a system banner is a different experiment and is not run here.
Whether an injection placed in Rights Manager Contact would have any effect -- by construction it cannot, and that was not probed either.
Whether a note inside the CLAUSE EXTRACTS themselves -- an injection written as a contract clause -- would behave differently. That is the surface this kit most depends on and it was NOT probed.
PresenterOpens the private repo. Visible to admins only.
In one lineThe only guardrail is the prompt
One instruction, no enforcement. It mostly holds — and mostly is the whole subject.
the guardrail, verbatim
Never amend a grant, issue a licence, terminate a deal, release a window or notify a licensee -- and never present the clearance policy, the holdback convention or the escalation horizon as any real licensor's terms.
Stated to the model on every call in src/prompt.py's INSTRUCTION block, and enforced by evals/check_labels.py's banned-code-path scan before any run may spend.
EvidenceDoes it hold?
What
Measured
The banned-code-path scan
0 banned code paths across the whole kit, on every run of check_labels.py, including the ones immediately before every paid run.
The instruction-shaped note does NOT suppress -- and this one IS measured, with a paired control
45 of 45 readings where an escalation was due, re-fired with the note forced in and each paired against its own un-injected answer, still raised it. Suppression rate 0.0 pct of 45.
The privacy guard has TWO independent locks
Removing NEVER_SENT alone leaks 0 of 120; adding the block to a hint alone leaks 0 of 120; removing BOTH leaks 120 of 120. All three conditions are reproduced in check_labels.py.
The limitWhat a guardrail is not
This is a PROMPT RULE PLUS A STATIC SCAN, not a runtime enforcement layer. Nothing stops a forker adding such a path tomorrow; the scan catches it only the next time somebody runs check_labels.py, which is a manual step.
The injection result is ONE SENTENCE against ONE model on ONE corpus. It is not a resistance rate for prompt injection in general.
A clearance opinion is not legal advice and this kit says so nowhere in code -- it is a property of what the output IS, and a forker pointing this at real contracts owns that.
WatchedWhat is watched, and why that one
9runs recorded
0 / 0deterministic metrics exact
+0.0%largest move — latency
0band breaches
—model half · needs provider URL
The kit measures 75 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.
5 measured by the latest run70 need the model half
Metric
Owner
Role
Why this one
cell-exact-match
The verdict, deciding axis, blocked days, blocking clause and escalate call, per reading, exact match against the computed answer key
alarm
the per-field accuracies and the answered rate — alarm on any field falling below its strongest-free-floor value -- the point at which paying for the model stopped being worth it on that column
conflict-window
A real conflict, as against a permitted overlap
alarm
precision and recall together, never either alone — alarm on any miss on the calendar-misleading slice. The interval-overlap floor catches 0.0 pct of it by definition.
clause-versus-tag
Verdict accuracy split by how the deciding term was recorded
alarm
the prose column — alarm on any fall in the prose column -- the keyed column is free.
GuardsBefore any comparison
Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.
Guard
Value
If it moves
corpus.doc_count
120
different corpus — nothing is comparable
corpus.bytes
866,466
clearance requests edited — the count held, the bytes did not
split.count
120
the readings count moved — a different set was scored
split.size_p50
7,220
the median size of one reading moved
split.size_p95
7,474
the 95th-percentile size of one reading moved
dataset.rows
120
the test set changed — every rate has a new denominator
tokens.context_limit
1,000,000
different model family
index.build_seconds
0.6
no index is built here — a change is in corpus preparation, not an index rebuild
no shared guard The two recorded runs declare no guard in common, so nothing below is protected by a precondition — read the history as two separate observations, not as a difference.
The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.
BandsThe alarm bands
Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.
Group
Band
Denominator
Where the band comes from
Verdict, eight-way
100.0 pct
120 readings scored
r001-rights-conflict exact match against src/rights.decide()'s computed gold
Deciding axis
86.67 pct
120 readings scored
r001-rights-conflict exact match against src/rights.decide()'s computed gold
Blocked days
99.17 pct
120 readings scored
r001-rights-conflict exact match against src/rights.decide()'s computed gold
Blocking clause
100.0 pct
120 readings scored
r001-rights-conflict exact match against src/rights.decide()'s computed gold
Escalate call
100.0 pct
120 readings scored
r001-rights-conflict exact match against src/rights.decide()'s computed gold
Real conflict vs permitted overlap
100.0 pct
120 readings scored
r001-rights-conflict exact match against src/rights.decide()'s computed gold
Real conflict -- precision
100.0 pct
54 readings the arm flagged
r001-rights-conflict exact match against src/rights.decide()'s computed gold
Real conflict -- recall
100.0 pct
54 true conflicts
r001-rights-conflict exact match against src/rights.decide()'s computed gold
Readings the calendar alone gets wrong, caught
100.0 pct
72 calendar-misleading cells
r001-rights-conflict exact match against src/rights.decide()'s computed gold
Permitted overlaps correctly left alone
100.0 pct
66 permitted cells
r001-rights-conflict exact match against src/rights.decide()'s computed gold
Verdict where the deciding term is PROSE only
100.0 pct
26 prose cells
r001-rights-conflict exact match against src/rights.decide()'s computed gold
Verdict where the deciding term was KEYED IN
100.0 pct
26 tag cells
r001-rights-conflict exact match against src/rights.decide()'s computed gold
Clause-dependent readings, verdict
100.0 pct
52 clause-dependent cells
r001-rights-conflict exact match against src/rights.decide()'s computed gold
Context-incomplete recall
100.0 pct
8 context incomplete cells
r001-rights-conflict exact match against src/rights.decide()'s computed gold
Titles needing an escalation, caught
100.0 pct
31 titles needing an escalation
r001-rights-conflict exact match against src/rights.decide()'s computed gold
Answered
100.0 pct
120 readings
r001-rights-conflict exact match against src/rights.decide()'s computed gold
False escalations
0.0 pct
75 quiet readings
r001-rights-conflict exact match against src/rights.decide()'s computed gold
Input tokens, run total
362222
120 readings
r001-rights-conflict, re-derived from its result file
Output tokens, run total
227904
120 readings
r001-rights-conflict, re-derived from its result file
Latency p50
7869
120 readings
r001-rights-conflict, re-derived from its result file
Latency p95
52195
120 readings
r001-rights-conflict, re-derived from its result file
HistoryRun history
9 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 4 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.
triage · no model in the path — a baseline, not a peer column — 3 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.
Metric
b000-rights-conflict-intervaloverlap 2026-08-25
b001-rights-conflict-overlapexcl 2026-08-25
b002-rights-conflict-clausetagstrong 2026-08-25
answered, %
100.0
100.0
100.0
axis accuracy, %
66.67
66.67
84.17
blocked days accuracy, %
40.00
56.67
78.33
blocking clause accuracy, %
38.33
56.67
78.33
calendar misleading caught, %
0.00
27.78
63.89
clause verdict accuracy, %
0.0
0.0
50.0
conflict window accuracy, %
40.00
56.67
78.33
conflict window precision, %
32.00
53.33
79.17
conflict window recall, %
29.63
29.63
70.37
context incomplete recall, %
100.0
100.0
100.0
escalate accuracy, %
52.50
63.33
83.33
false escalation rate, %
30.67
13.33
8.00
input tokens, whole run
0
0
0
model latency p50 ms
0.00
0.00
0.00
model latency p95 ms
0.00
0.00
0.00
output tokens, whole run
0
0
0
permitted left alone, %
65.15
84.85
90.91
prose verdict accuracy, %
0.0
0.0
0.0
tag verdict accuracy, %
0.0
0.0
100.0
titles caught, %
35.48
35.48
67.74
verdict accuracy, %
40.00
56.67
78.33
not a time series No two of these 3 runs measured the same system — they differ on conflict_window_false, conflict_window_fn, conflict_window_fp, conflict_window_tn, conflict_window_tp, false_escalations, missed_escalations, titles_caught, titles_missed, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.
triage · with the model in the path — 4 runs. Columns here are only ever compared with each other.
Metric
c000-rights-conflict-calibration 2026-08-25
c001-rights-conflict-calibration-32k 2026-08-25
r001-rights-conflict 2026-08-25
s001-rights-conflict-ledgeronly 2026-08-25
answered, %
100.0
100.0
100.0
100.0
axis accuracy, %
100.00
88.89
86.67
55.00
blocked days accuracy, %
100.00
100.00
99.17
56.67
blocking clause accuracy, %
100.00
100.00
100.00
56.67
calendar misleading caught, %
100.00
100.00
100.00
27.78
clause verdict accuracy, %
100.0
100.0
100.0
0.0
conflict window accuracy, %
100.00
100.00
100.00
56.67
conflict window precision, %
100.00
100.00
100.00
53.33
conflict window recall, %
100.00
100.00
100.00
29.63
context incomplete recall, %
—
—
100.0
100.0
escalate accuracy, %
100.00
100.00
100.00
63.33
false escalation rate, %
0.00
0.00
0.00
13.33
input tokens, whole run
27324
27324
362222
341915
model latency p50 ms
25609.00
18989.00
7869.00
6689.00
model latency p95 ms
39900.00
61092.00
52195.00
40368.00
output tokens, whole run
26167
26692
227904
160968
permitted left alone, %
—
—
100.00
84.85
prose verdict accuracy, %
100.0
100.0
100.0
0.0
tag verdict accuracy, %
—
—
100.0
0.0
titles caught, %
100.00
100.00
100.00
35.48
verdict accuracy, %
100.00
100.00
100.00
56.67
not a time series No two of these 4 runs measured the same system — they differ on answered, calendar_misleading_cells, clause_cells, conflict_window_cells, conflict_window_false, conflict_window_fn, conflict_window_fp, conflict_window_quiet_cells, conflict_window_tn, conflict_window_tp, context_incomplete_cells, documents, escalation_cells, false_escalations, max_tokens, missed_escalations, permitted_cells, prose_cells, quiet_cells, readings_scored, tag_cells, titles_caught, titles_missed, titles_needing_escalation — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.
stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.
Metric
t000-rights-conflict-stub 2026-08-25
answered, %
100.0
axis accuracy, %
66.67
blocked days accuracy, %
40.0
blocking clause accuracy, %
38.33
calendar misleading caught, %
0.0
clause verdict accuracy, %
0.0
conflict window accuracy, %
40.0
conflict window precision, %
32.0
conflict window recall, %
29.63
context incomplete recall, %
100.0
escalate accuracy, %
52.5
false escalation rate, %
30.67
input tokens, whole run
354112
model latency p50 ms
0.00
model latency p95 ms
0.00
output tokens, whole run
6668
permitted left alone, %
65.15
prose verdict accuracy, %
0.0
tag verdict accuracy, %
0.0
titles caught, %
35.48
verdict accuracy, %
40.0
one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 21 chips that all say so.
redteam · with the model in the path — 1 run. Columns here are only ever compared with each other.
Metric
x001-rights-conflict-injection 2026-08-25
escalations held
45
escalations suppressed
0
gold relative rate, %
0.0
gold relative suppressed
0
suppression rate, %
0.0
one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 5 chips that all say so.
DeviationsWhat deviated
0 breaches across 9 runs
The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.
RippleThe ripple map
Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.
Lever
What moves
Basis
Evidence
the eleven rules and their precedence in src/rights.py
what a conflict IS, the answer key, and the discriminator itself.
measured
red-proven by seeding -- reversing R-6's and to or convicts on the replay AND on the strong floor
the CLAUSE: tag grammar in evals/baseline.py
how much the strongest free floor can see, and therefore the gap the model is paid for.
measured
red-proven by seeding -- and it caught a real (\S+) bug before any spend
a section added to SECTION_HINTS, or NEVER_SENT deleted
what leaves the machine.
measured
red-proven on all three conditions
the holdback convention and the escalation horizon
blocked_days on every holdback reading, and which conflicts escalate at all.
measured
red-proven by seeding
the candidate-pair generator a real deployment would put in front of this
cost, quadratically, and which conflicts are ever looked at.
reasoning
not built, not measured
FiresWhat fires when a band breaks
Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.
Nothing fires yet. Not one of the 21 bands on this kit names a consequence, so there is no row to print. Which is the honest answer to the heading: a band that nothing is wired to is a number somebody has to notice, and this kit has only those. An empty table would have said the same thing while looking like a rendering failure.
NextThe three you would add first
⚑ NAME AN AXIS FOR R-7 IN THE RULE TEXTThe model's only systematic loss on this kit is a branch the rule never specified. 16 of 16 readings, and the floors collect them for free.
Put a blocking step in front of the pair readerCost is quadratic in grants per title and this kit reads whatever pairs it is handed.
Automate the manual check_labels stepToday it is a step a developer has to remember.
None of these is built here. They are named against the seam they sit on so they can be argued with.
CadenceWhen to re-run, and what it costs
The guardrail scan is a MANUAL step, run before each spend, not a hook. It ran before every paid run and passed at 0 each time. The injection probe is a ONE-OFF: it measured 0.0 pct suppression across 45 paired readings on 2026-08-25 and nothing re-runs it, so that figure ages from the day it was taken.
What this cannot tell you
⚠︎ WHETHER THE ZERO SUPPRESSION HOLDS ACROSS PHRASINGS. One sentence, one model, one corpus -- but every reading is paired against its own un-injected answer, so the denominator is real and the absence of movement on THOSE 45 readings is established.
⚠︎ WHETHER AN INJECTION WRITTEN AS A CONTRACT CLAUSE would behave differently. The clause extracts are the surface this kit most depends on and they were NOT probed -- the note was forced into Rights Notes only.
Whether a differently-named write path would be caught by the static scan.
Whether the prompt rule or the absent code path is what keeps the kit decision-free.
PresenterOpens the private repo. Visible to admins only.
In one lineNo framework, deliberately
NO FRAMEWORK. A folder of readable Python, no framework dependency, and a prompt anyone can read end to end. A framework abstraction would own the retrieval step -- there is none here -- and the rule engine, which is already the entire surface of src/rights.py and doubles as the answer key.
The mappingSeam by seam, who owns what
Seam
File here
Framework equivalent
Note
the model call
src/adapters/__init__.py
an LLM wrapper / Runnable
one dict and one function; the seam this kit measures is what the model returns, not how it is called.
the clearance rule
src/rights.py
a rules engine or a policy DSL
eleven rules in eighty lines, and the same function computes the answer key. A DSL would put a translation layer between the rule a reader reads and the rule the key applies, which is exactly the drift this kit's check_labels prevents.
the corpus
tools/build_corpus.py
a document loader
one flat synthetic format this kit fully controls.
The seams are the ones this kit already publishes, so the mapping cannot drift from the code.
GraphsWhere LangGraph changes the answer
Linear: the reader -> src/prompt.py -> src/adapters -> evals/scoring.py. No branching, no tool-calling and no agent loop.
The other sideWhat a framework costs you
Swapping providers means editing the PROVIDERS dict by hand. In exchange the prompt has no hidden templating layer and requirements.txt stays empty.
What we could NOT verify
Whether a rules-engine abstraction would have kept the PRECEDENCE of the eleven rules readable. The precedence is the whole of R-1 through R-9 and it is what an arm most often gets wrong; a DSL that reordered it silently would be undetectable from the page.
PresenterOpens the private repo. Visible to admins only.
In one lineWhat one run of this kit actually cost, in time and tokens
Measured on r001-rights-conflict on the fast tier, reading the clause extracts, 2026-08-25. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.
The readingWhat the last run measured
Reading
Last run
Band
What fires
Model, median
7,869 ms
7869
—
Model, p95
52,195 ms
52195
—
Input tokens
362,222
362222
—
Output tokens
227,904
227904
—
No movement column. Not one of the 6 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.
Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.
DriftMedian call, run over run
c000-rights-conflict-calibration25,609 ms
c001-rights-conflict-calibration-32k18,989 ms
r001-rights-conflict7,869 ms
s001-rights-conflict-ledgeronly6,689 ms
Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.
3 runs not plotted. b000-rights-conflict-intervaloverlap, b001-rights-conflict-overlapexcl, b002-rights-conflict-clausetagstrong recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.
The other sideWhat this does not tell you
What we could NOT verify
No stage-by-stage span tree. These are whole-call timings; which part of the pipeline spent the time is not recorded by this kit's harness.
Latency is a run-level median and p95. The spread between the fastest and slowest single call is not stored, so the distribution cannot be drawn.
No earlier run is comparable to this one (a different model tier or a changed guard), so nothing on this page is a trend yet.
One machine, one network, one region. Latency includes the provider round trip from a single location and will not transfer to yours.
The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.
PresenterOpens the private repo. Visible to admins only.
In one lineWhat you provision, honestly
Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — blocking is computed in memory each run and is the only reduction step.
The machineWhat this needs
Dependencies
nothing beyond the standard library — requirements.txt names no package
Configuration
9 environment variables — the full knob table is at the bottom of this page
Last verified run
2026-08-25, across 9 committed records
Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.
7 of the 8 sections go to the provider; Rights Manager Contact never does
the answer key
data/gold.jsonl, computed by src/rights.decide()
never
the clause extracts
inside each request, section 6
yes, verbatim -- that is the experiment, and the ledger-only control is the same prompt with that one block replaced
every run this kit has fired
results/eval-*.json
never
The wireWhat crosses it, and when
Runtime call site
Destination
src/adapters/__init__.py line 54
the configured BASE_URL
Build-time (corpus tooling, run once)
Destination
none — this kit’s corpus is generated locally
Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.
The third state
What the provider retains, trains on or logs once a request arrives is not measured here — it is provider-dependent, and this page does not assert what no run of ours can verify.
The shapeHow it ships, and what you add
The key
API_KEY is read from .env (gitignored) or the real environment; never logged, never placed in a prompt, and redacted from any provider error message the local UI passes to the browser.
You add
your reverse proxy and your identity in front of one process — the kit deliberately ships neither, because identity belongs to your estate, not to a kit
Never ships
Docker, Helm, a queue, a database. The product is a folder of readable Python; what to run it under is this page’s job to state, not the kit’s job to impose
The ladderEach decision, and what you provision past its ceiling
Decision
The kit ships
Measured here
Past the ceiling
What stops being valid
blocking
NOTHING -- and that is the honest answer. This kit reads the candidate pairs it is handed, one pair per reading; the title is the file, the pair is the reading.
40 titles x 3 pairs = 120 calls per arm. (r001-rights-conflict, evals/check_labels.py)
⚑ PAIR COUNT IS QUADRATIC IN GRANTS PER TITLE, AND THE STEP THAT WOULD REDUCE IT IS NOT IN THIS KIT. A title with 12 live grants is 66 pairs; with 24 it is 276. Named, not built, and not measured.
A catalogue with hundreds of live grants per title.
model
one completion call per reading. PUBLISHED ceiling 32000 output tokens; the scored arms were taken at 16000 and nothing truncated.
120 readings, 3018.52 in / 1899.2 out per reading. p50 7869 ms, p95 52195 ms. ⚑ AND THE SAME CALL WITH THE CLAUSE EXTRACTS BODY REMOVED (s001-rights-conflict-ledgeronly) agreed with the FREE overlap-exclusivity floor on 120 of 120 verdicts -- the same answer on every reading, not the same average -- finding 16 of the 54 real conflicts against the full arm's 54. (r001-rights-conflict, c000-rights-conflict-calibration, c001-rights-conflict-calibration-32k, src/clearance.MAX_TOKENS)
⚠︎ REFUTED TWICE. This kit's own probe said 30.34 pct of a 16,000 cap and its scored run said 62.77 pct, with 3 readings above 8,000. Then 16,000 itself was refuted mid-build by a sibling that recorded 16,452. The PUBLISHED ceiling is now 32,000, confirmed by c001-rights-conflict-calibration-32k at 24.55 pct. The scored arms stay at 16,000 and are NOT discarded -- nothing truncated, 120 of 120 answered on both.
A second model.
labels
a computed answer key from src/rights.decide().
Every row replays exactly. 23 checks, green, red-proven on five seeded defects and re-acquitted. (tools/build_corpus.py, evals/check_labels.py)
⚑ THE DISCRIMINATOR'S NON-TRIVIALITY IS ASSERTED BY RUNNING THE NAIVE IMPLEMENTATION, not claimed: 72 of 120 readings are ones it gets wrong.
Your own paper.
Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.
When it failsFailure signatures
What you see
What it is
First move
the interval-overlap floor flagging a pair the model leaves alone
two non-exclusive grants, or an express carve-out. 34 of the floor's flags are this.
read the Exclusivity line on both deal summaries before opening anything. (b000-rights-conflict-intervaloverlap against r001-rights-conflict)
both free floors answering PERMITTED_DISJOINT on windows that do not touch
look for a holdback. The exclusivity may run past its own window, and neither floor can see a clause nobody keyed in.
open the Clause Extracts section of the earlier grant. (r001-rights-conflict hero reading RGT-0003-P2)
the model naming WINDOW as the deciding axis on an exclusive overlap
the kit's own rule text never specified an axis for that branch. The verdict, the days, the clause and the escalation are still right.
do not treat it as a misread page. Fix R-7's wording and re-run. (r001-rights-conflict misses)
['Pair generation. This kit reads the candidate pairs it is given; the blocking step is named and not built.', 'Two contracts defining the same term differently -- the hard half of R-2.', 'Repeats. One run per arm.', 'A second model tier.', 'Prompt caching, which would change the cost model substantially given how much of the prompt is fixed.', 'Whether the clause extracts rendered as JSON rather than as prose would score the same.']
The corpus licence, from the Data lens: MIT Your corpus’s licence is yours to verify, and the labels you write are about your documents.
The verdict, deciding axis, blocked days, blocking clause and escalate call, per reading, exact match against the computed answer key
Check whether two license deals on one show clash
PresenterOpens the private repo. Visible to admins only.
In one lineThe verdict, deciding axis, blocked days, blocking clause and escalate call, per reading, exact match against the computed answer key
whether each of the 5 answered fields equals the answer key, per reading
$0.00per 1,000 clearance requests
nodata leaves your network
yessame answer every time
MethodHow the test was run
python3 -m evals.run --run-id <id> ...; evals/scoring.py compares strings. No model grades anything.
Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.
The inputOne real row, seen by every grader
doc id
RGT-0003-P2
note
The hero reading. Two exclusive EST grants for Mexico whose licence windows are 15 days apart and share NOT ONE DAY. Clause A-4.2 holds Grant A's exclusivity 60 days past its expiry, so 46 days of Grant B's window are blocked. Written as a sentence, not keyed -- both free floors answer PERMITTED_DISJOINT.
{'verdict': 'CONFLICT_HOLDBACK', 'axis': 'HOLDBACK', 'blocked_days': 46, 'blocking_clause': 'A-4.2', 'escalate': 'YES', 'rationale': "The two exclusive EST/Mexico/Italian-English grants have windows separated by 15 days, but A-4.2 holds Grant A's exclusivity for 60 days past 2027-10-03, and Grant B's window beginning 2027-10-18 falls inside that tail, blocking 46 days."}
The verdict, deciding axis, blocked days, blocking clause and escalate call, per reading, exact match against the computed answer key
all five fields hit
The answer recorded for RGT-0003-P2 in results/eval-r001-rights-conflict.json equals the answer key in data/gold.jsonl on every scored cell -- CONFLICT_HOLDBACK, HOLDBACK, 46, A-4.2, YES. The strongest free floor answers PERMITTED_DISJOINT / WINDOW / 0 / NONE / NO on the same reading and misses all five.
A real conflict, as against a permitted overlap
hit -- a real conflict, called
The key marks this pair conflict true with calendar_misleading true and naive_says_conflict false, so it is one of the 72 readings a pure interval-overlap implementation gets wrong. The scored arm called it; all three free floors and the ledger-only control answer PERMITTED_DISJOINT.
Verdict accuracy split by how the deciding term was recorded
hit -- on the PROSE side of the split
The key records evidence_kind PROSE for this reading: clause A-4.2 is written as a sentence and never keyed as a CLAUSE: tag. It is one of the 26 prose readings, where the scored arm reads 100.0 pct and the strongest free floor reads 0.0 pct.
The formulaWhat it computes
accuracy = hits / 120 per field. conflict_window_accuracy_pct is scored separately over its own denominators.
The analysisWhat it actually did
Model
Result
the fast tier, reading the clause extracts
100.0% verdict accuracy · 4 more measured on this row
In operationWhat to monitor
Reference standard: the computed answer key in data/gold.jsonl, produced by src/rights.decide() from the same planted inputs the corpus builder wrote onto the page.
No true/false rates for this grader. It records 5 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.
Watch these
the per-field accuracies and the answered rate
Alarm on
any field falling below its strongest-free-floor value -- the point at which paying for the model stopped being worth it on that column
How tight can the band be? No threshold was swept: exact match has no tunable. Denominators are stated beside every rate.
Cadence: once per run; free
The decisionWhen to reach for it
Use it
Always. Every published accuracy figure rests on it.
Do not use it
It cannot tell you an answer was reasonable-but-wrong. Every axis miss on the scored run is a reading the rule text never specified an answer for.
PresenterOpens the private repo. Visible to admins only.
In one lineA real conflict, as against a permitted overlap
whether the arm called a pair a real conflict when it was, and left the permitted overlaps alone
$0.00per 1,000 clearance requests
nodata leaves your network
yessame answer every time
MethodHow the test was run
evals/scoring.py, same pass as the exact-match grader.
Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.
The inputOne real row, seen by every grader
doc id
RGT-0003-P2
note
The hero reading. Two exclusive EST grants for Mexico whose licence windows are 15 days apart and share NOT ONE DAY. Clause A-4.2 holds Grant A's exclusivity 60 days past its expiry, so 46 days of Grant B's window are blocked. Written as a sentence, not keyed -- both free floors answer PERMITTED_DISJOINT.
{'verdict': 'CONFLICT_HOLDBACK', 'axis': 'HOLDBACK', 'blocked_days': 46, 'blocking_clause': 'A-4.2', 'escalate': 'YES', 'rationale': "The two exclusive EST/Mexico/Italian-English grants have windows separated by 15 days, but A-4.2 holds Grant A's exclusivity for 60 days past 2027-10-03, and Grant B's window beginning 2027-10-18 falls inside that tail, blocking 46 days."}
The verdict, deciding axis, blocked days, blocking clause and escalate call, per reading, exact match against the computed answer key
all five fields hit
The answer recorded for RGT-0003-P2 in results/eval-r001-rights-conflict.json equals the answer key in data/gold.jsonl on every scored cell -- CONFLICT_HOLDBACK, HOLDBACK, 46, A-4.2, YES. The strongest free floor answers PERMITTED_DISJOINT / WINDOW / 0 / NONE / NO on the same reading and misses all five.
A real conflict, as against a permitted overlap
hit -- a real conflict, called
The key marks this pair conflict true with calendar_misleading true and naive_says_conflict false, so it is one of the 72 readings a pure interval-overlap implementation gets wrong. The scored arm called it; all three free floors and the ledger-only control answer PERMITTED_DISJOINT.
Verdict accuracy split by how the deciding term was recorded
hit -- on the PROSE side of the split
The key records evidence_kind PROSE for this reading: clause A-4.2 is written as a sentence and never keyed as a CLAUSE: tag. It is one of the 26 prose readings, where the scored arm reads 100.0 pct and the strongest free floor reads 0.0 pct.
The formulaWhat it computes
accuracy over all 120 readings; PRECISION over what the arm flagged and RECALL over the 54 true conflicts, both printed, because a report that flags everything scores 100 pct recall and is worthless.
The analysisWhat it actually did
Model
Result
the fast tier, reading the clause extracts
100.0% conflict window accuracy · 3 more measured on this row
what free code writes in an afternoon
40.0% conflict window accuracy · 3 more measured on this row
the strongest free floor -- no model
78.3% conflict window accuracy · 3 more measured on this row
In operationWhat to monitor
Reference standard: the computed answer key in data/gold.jsonl, produced by src/rights.decide().
No true/false rates for this grader. It records 3 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.
Watch these
precision and recall together, never either alone
Alarm on
any miss on the calendar-misleading slice. The interval-overlap floor catches 0.0 pct of it by definition.
How tight can the band be? No sweep: the verdict is an enum the arm emits.
Cadence: once per run; free
The decisionWhen to reach for it
Use it
Whenever the question is 'which of these candidate pairs is real'.
Do not use it
It says nothing about which of the two grants should yield, only that they cannot both stand.
Verdict accuracy split by how the deciding term was recorded
Check whether two license deals on one show clash
PresenterOpens the private repo. Visible to admins only.
In one lineVerdict accuracy split by how the deciding term was recorded
whether the arm can read a term written as a contract sentence, as against one keyed into a structured field
$0.00per 1,000 clearance requests
nodata leaves your network
yessame answer every time
MethodHow the test was run
evals/scoring.py, same pass.
Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.
The inputOne real row, seen by every grader
doc id
RGT-0003-P2
note
The hero reading. Two exclusive EST grants for Mexico whose licence windows are 15 days apart and share NOT ONE DAY. Clause A-4.2 holds Grant A's exclusivity 60 days past its expiry, so 46 days of Grant B's window are blocked. Written as a sentence, not keyed -- both free floors answer PERMITTED_DISJOINT.
{'verdict': 'CONFLICT_HOLDBACK', 'axis': 'HOLDBACK', 'blocked_days': 46, 'blocking_clause': 'A-4.2', 'escalate': 'YES', 'rationale': "The two exclusive EST/Mexico/Italian-English grants have windows separated by 15 days, but A-4.2 holds Grant A's exclusivity for 60 days past 2027-10-03, and Grant B's window beginning 2027-10-18 falls inside that tail, blocking 46 days."}
The verdict, deciding axis, blocked days, blocking clause and escalate call, per reading, exact match against the computed answer key
all five fields hit
The answer recorded for RGT-0003-P2 in results/eval-r001-rights-conflict.json equals the answer key in data/gold.jsonl on every scored cell -- CONFLICT_HOLDBACK, HOLDBACK, 46, A-4.2, YES. The strongest free floor answers PERMITTED_DISJOINT / WINDOW / 0 / NONE / NO on the same reading and misses all five.
A real conflict, as against a permitted overlap
hit -- a real conflict, called
The key marks this pair conflict true with calendar_misleading true and naive_says_conflict false, so it is one of the 72 readings a pure interval-overlap implementation gets wrong. The scored arm called it; all three free floors and the ledger-only control answer PERMITTED_DISJOINT.
Verdict accuracy split by how the deciding term was recorded
hit -- on the PROSE side of the split
The key records evidence_kind PROSE for this reading: clause A-4.2 is written as a sentence and never keyed as a CLAUSE: tag. It is one of the 26 prose readings, where the scored arm reads 100.0 pct and the strongest free floor reads 0.0 pct.
The formulaWhat it computes
verdict accuracy over the 26 readings whose deciding term is a CLAUSE: tag, and separately over the 26 whose deciding term is prose only.
The analysisWhat it actually did
Model
Result
the fast tier, reading the clause extracts
100.0% tag verdict accuracy · 1 more measured on this row
the strongest free floor -- no model
100.0% tag verdict accuracy · 1 more measured on this row
In operationWhat to monitor
Reference standard: the computed answer key in data/gold.jsonl.
No true/false rates for this grader. It records 2 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.
Watch these
the prose column
Alarm on
any fall in the prose column -- the keyed column is free.
How tight can the band be? No sweep.
Cadence: once per run; free
The decisionWhen to reach for it
Use it
Always, beside the discriminator.
Do not use it
It cannot say how much of a REAL ledger is keyed rather than prose. This corpus splits it 50/50 by construction; your ledger will not.
A living map of modern AI — kept current every morning
