The business caseThe problem this solves

A loss file lands and somebody has to answer one question before it closes: is there a party other than the insured that a recovery could run against, and what in the file says so? A file tells you WHO DID WHAT, and that is the one thing only a reader can supply. Everything after it -- whether a waiver covers that party, whether they are named on the same schedule as the party claiming, whether the amount clears the threshold, whether the period has run -- is a role lookup and two comparisons. Deciding a recovery is mostly not a reading problem. It is a clause lookup with a reading problem in front of it, and the reading problem is where the file hides the answer: on nine of these sixty files the narrative says the cause was never established and the attached engineer's report names the party who installed the failed part. The reviewer's manual pass over one loss file against the recovery terms -- reading the narrative, the parties block and the attachments for who is responsible, then checking the waiver, the named-insured schedule, the threshold and the limitation period -- before a person confirms the decision. It replaces neither the confirmation nor the recovery action; there is no endpoint here that opens one.

The inputThe actual loss files

The corpus is 60 loss files, 0.07 MB (txt 60). The cases are the ones that make this decision hard, and none of them can be scraped. A contractor who plainly caused the loss and against whom recovery has already been waived -- the file's own terms bar it while every trigger word in the narrative says otherwise. A file whose narrative says the cause was never established and whose attached engineer's report names the party who installed the failed part. Two accounts on one file that contradict each other, each side blaming the other, where there is a party named and there is no answer. A second vehicle that left before details were exchanged -- a third party is responsible and there is nobody to pursue, which is not the same as nobody being responsible. A file where the terms need to know whether an agreement was executed before the loss and the file DOES NOT RECORD IT, where reading absent as 'no' opens a waived recovery. And a decoy: two files that list the supplier who delivered the load first, so an arm that takes the first party who is not the insured names somebody the schedule does not cover. All five clauses are in force on EVERY file, deliberately -- a reader who checks whether a file MENTIONS a waiver learns nothing here, because every file mentions one; the question is always whether the condition is met. A quarter of the set is clean, because a corpus that is all traps measures a different job from the one the reviewer has.

LOSS FILE LF-0001 -- Impact damage -- yard

FILE FACTS
  Reference                     RF-1001
  Date of loss                  2024-02-03
  Amount claimed                USD 21,500
  Location                      the north yard shutter, the Ockley depot
  Named insureds on schedule    Brantley Marine plc
  Services agreement            none in place

INCIDENT NARRATIVE

A member of the insured's staff reported that the north yard shutter had been struck at some point during the shift. The site was in normal operation and the area was not closed to vehicles.

Setterfield & Co Ltd's driver reversed a box van into the north yard shutter while turning in the yard, and Setterfield & Co Ltd has confirmed in writing that the vehicle and its driver were under its control at the time.

The damage is confined to the north yard shutter. Photographs taken the same day are on the file and the amount claimed is the repair quoted by the insured's own contractor.

PARTIES ON THE FILE

  Brantley Marine plc           the insured itself
  Setterfield & Co Ltd          a supplier or visiting party with no services agreement in place

ATTACHMENTS

  none

The outcomeWhat a good result looks like

A recovery decision a person confirms instead of a narrative and a set of terms read side by side: the verdict, the party a recovery would run against spelled as the file spells it, the clause that bars it where one does, and ONE SENTENCE COPIED VERBATIM out of the file that establishes the third party's responsibility -- located in the file and highlighted where it was found, never paraphrased.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Files where the responsible party and the sentence that says so are both in the narrative, on a book with a clean register — the free rules floor alone
  On the six plain_third_party and four rear_impact_clear files it gets the verdict and the party right 10 of 10 -- it only loses the citation, quoting an earlier trigger-carrying sentence. And it gets EVERY clause free: 5 of 5 waiver_bars, 4 of 4 waiver_present_not_applicable, 3 of 3 below_threshold, 3 of 3 missing_bar_fact and 3 of 3 narrative_implies_bar_language, all-correct, because the clause work is the terms engine and the terms engine costs nothing.
• Files that carry the finding in an attachment while the narrative says the cause was never established — the model, rechecked
  9 of 9 against the floor's 0 of 9, and the floor's zero is STRUCTURAL: it reads the narrative only and cannot be tuned to score anything else. On six of those files the floor's answer is a missed recovery and on the other three it is the right verdict for a reason the file does not contain. This row is the whole case for the call.

And where nothing here is good enough:

• Files where two accounts contradict each other and the right answer is to refuse — NEITHER ARM, AND THIS KIT SAYS SO. Refer them by rule.
  conflicting_accounts is the ONE case where both arms score 0 of 4 all-correct. The floor answers a confident RECOVERABLE against the party named first. The model calls the verdict right 4 of 4 and cites a real sentence 4 of 4 and then NAMES A PARTY ANYWAY on all four -- its entire party_invented count and 4 of its 5 misses. The kit has no control for it: recheck_overrides is 0 and the terms engine checks clauses, not whether a name belongs.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-31. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-31 — r001-recovery-flag. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — ⚠︎ PARTIALLY EVIDENCED, AND THE REST IS NOT MEASURED. What is on disk and checkable: the 60 files, the register, RT-2026 in both renderings, the answer key with its citation spans, and TWO COMMITTED KEYLESS RUNS -- results/eval-b000-recovery-flag-rules.json (model 'free-floor:rules', usd null, 29 of 60) and results/eval-t000-recovery-flag-stub.json (model 'stub', usd null, 29 of 60, largest reply 146 tokens) -- neither of which could have been produced with a provider in the loop. The scored run's cache (results/cache-r001-recovery-flag.jsonl) ships too, so the model columns render without a call. NOT MEASURED: no timed fresh-clone run was recorded for this kit -- no HTTP-200-on-every-endpoint sweep of the board and no stopwatch on tools/build_corpus.py or evals/check_labels.py. The claim here is 'the artefacts that make the free half work are committed', not 'a cold clone was timed'.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. one loss file end to end -- one file, one model call, including provider-side reasoning tokens, on a shared connection. The terms engine adds nothing measurable: it is pure Python over the reply. ⚠︎ NOT AN SLA and NOT A SPREAD: this credential is shared with sibling kits, these are wall-clock observations from ONE run, and the band is wide -- fastest call 3233 ms, slowest 80958 ms. 60 files ran in 164.1 wall seconds at 5 workers (EVAL_WORKERS default in evals/run.py); the free rules floor and the terms engine answer all 60 with no network at all and the floor run records 0.0 wall seconds.

Current processWhat it replaces

The reviewer's manual pass over one loss file against the recovery terms -- reading the narrative, the parties block and the attachments for who is responsible, then checking the waiver, the named-insured schedule, the threshold and the limitation period -- before a person confirms the decision. It replaces neither the confirmation nor the recovery action; there is no endpoint here that opens one.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one recovery decision is

No train/test split, because nothing is trained or tuned, and no segmentation, because one loss file goes whole into one call (the sizes here are the files' own bytes; min 870, max 1,640). The unit every rate stands on is ONE RECOVERY DECISION, one per file: 60 files, 60 decisions, and every arm is scored over all 60 whether or not it answered. 15 files are clean; 45 plant exactly one thing built to be got wrong, across 15 named cases: attachment_only 6, attachment_only_barred 3, plain_third_party 6, sole_fault_insured 5, waiver_bars 5, rear_impact_clear 4, no_third_party_natural 4, co_insured_bars 4, conflicting_accounts 4, waiver_present_not_applicable 4, below_threshold 3, missing_bar_fact 3, narrative_implies_bar_language 3, time_barred 3, unnamed_third_party 3.

SetupWhat the setup figure measured

THERE IS NO INDEX. There is nothing to retrieve from: one loss file goes whole into one prompt with the terms, at a mean of 2,822.4 input tokens. The whole free half is the fraction of a second the floor run records -- results/eval-b000-recovery-flag-rules.json carries wall_seconds 0.0 for answering and scoring all 60 files with no key and no network. ⚠︎ 0.0 IS THE RECORDED VALUE, NOT A ROUNDED MEASUREMENT: the harness writes wall_seconds only for the model arm's calls and the free arm makes none, so read it as 'no network time', not as a benchmark.

LicenceLicence

MIT, with the corpus, the register and the terms generated in-process -- there is no third-party data in this kit to licence.

Bring your ownBring your own loss files

Drop your own loss files as .txt into data/corpus/, add one row per file to data/files.json (id, loss_date, loss_amount, currency, named_insureds, parties with a role from src/verdicts.py::ROLES, and agreement_executed_before_loss as true / false / NULL FOR 'NOT RECORDED' -- the third state is load-bearing, because absent is not 'no'), and write data/gold.jsonl with the verdict, the party and the citation. The span is computable: src/citation.py::span_of(norm(text), sentence). Replace data/terms.json and data/terms.md with your own wording -- clause ids, threshold and period are read, never hard-coded -- and add a test function to src/terms.py::TESTS for any condition the four here do not cover. Nothing else changes; the UI and the free floor work immediately, and without a gold.jsonl of your own the eval has nothing to score against.

⚠︎ And what stops being true when you do: ⚠︎ A REAL LOSS FILE NAMES THE PARTIES TO A LIVE DISPUTE AND QUOTES STATEMENTS TAKEN FROM PEOPLE, AND THE WHOLE FILE REACHES YOUR CONFIGURED PROVIDER VERBATIM -- the third party's name, the driver's statement, the engineer's report and anything said about anybody. There is no redaction step in this kit and the seam that protects the register (data/files.json never leaves the machine) protects nothing in the file's own text, because the text is the input. That is a disclosure decision for whoever owns the file and not one a kit can make: run it against an extract you are permitted to disclose, or against a local model. And every measured rate here is a claim about fifteen planted cases in one generated register -- a real book of files contains kinds this set does not.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 11,956 characters assembled, 5,547 are instructions — 46% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

The exact six parts sent for LF-0001, the first file of run r001-recovery-flag, replayed from the kit's own src/prompt.py build(). Trustworthy as the prompt that was sent because the replayed part names and character counts (1,752 / 3,630 / 1,600 / 1,156 / 1,179 / 2,639 = 11,956) are byte-for-byte the decomposition the run itself recorded in results/eval-r001-recovery-flag.json under prompt_parts and prompt_chars_total. Only the file part varies across the corpus: replaying all 60 gives a whole-prompt range of 11,673-12,443 characters, mean 11,949.

Full promptThe prompt in full

You are reviewing one loss file and deciding a single question: is there a party other than
the insured that a recovery could run against, and what in the file says so? Your output is the
recovery decision a reviewer confirms -- so that the reviewer confirms a decision instead of
reading a narrative and a set of terms side by side.

How to read the file:

- The INCIDENT NARRATIVE, the PARTIES ON THE FILE block and every ATTACHMENT are all part of the
  file and all of them count. A file often establishes responsibility only in an attachment while
  its narrative says the cause was never settled; an attachment is not a lesser document.
- The PARTIES block records each party's RELATIONSHIP to the insured. That relationship is a fact
  about the arrangement between them, never about fault: a party carrying out work under an
  agreement can be plainly responsible for the loss and still be a party the terms bar recovery
  against.
- The FILE FACTS block is the authority on the date of loss, the amount claimed, who is named as
  an insured on the schedule, and whether a services agreement was executed before the loss. Where
  it says a fact is NOT RECORDED, that is not the same as "no" -- it is a fact the file does not
  have, and it is a reason to answer NEEDS-INVESTIGATION rather than to assume either way.
- Apply the recovery terms as written, including the order they are applied in and the threshold
  and period they state.
- Name the party even where a clause bars recovery against them. Who the recovery would have run
  against and whether it may be opened are two separate answers and this file wants both.
- Give a confidence between 0 and 1 for your verdict.

Reply with JSON and nothing else, in the shape given at the end.

THE RECOVERY TERMS, as they stand:

# RT-2026 — terms governing recovery from a third party

These terms decide one thing: once a loss file has established that somebody other than the
insured caused or contributed to a loss, may a recovery be opened against them?

**All five clauses below are in force on every file in this corpus.** A file that mentions a
waiver is not a file where a waiver applies — the question is always whether the clause's
condition is met by the facts recorded on that file.

## R-1 — Right of recovery

Where a party other than the insured has by its act or omission caused or contributed to a loss,
recovery may be pursued against that party for the amount paid, subject to the clauses that
follow.

## R-2 — Waiver of recovery: work under an executed agreement

No recovery shall be pursued against a party carrying out work at the loss location under a
services agreement executed before the date of loss. Where the file does not record whether such
an agreement was executed before that date, the question is referred and no recovery is opened in
the meantime.

**The condition has two limbs and both must be met.** The party must be carrying out work at the
loss location under a services agreement with the insured — a supplier delivering goods, a
visiting party, or the operator of another vehicle is not carrying out work under one — and that
agreement must have been executed before the date of loss. A party plainly at fault is still a
party this clause bars recovery against.

## R-3 — Recovery against a party insured on the same schedule

No recovery shall be pursued against a party named as an insured on the same schedule as the
party claiming.

The named insureds are listed on each file. A party that appears there is not a third party for
the purpose of these terms however the loss happened.

## R-4 — Threshold below which recovery is not pursued

No recovery shall be pursued where the amount claimed on the file is below USD 2,500.

## R-5 — Period within which recovery must be opened

No recovery shall be opened more than 3 years after the date of loss.

## Order of application

The clauses are applied in the order **R-2, R-3, R-4, R-5** and the first one whose condition is
met is the one recorded on the file. Where more than one would bar the same recovery the outcome
is the same and the recorded clause is the first.

## What is not decided here

These terms say nothing about who caused a loss. That is read off the file — the incident
narrative, the parties recorded on it, and anything attached to it — and it is the one thing
these clauses take as given.

*RT-2026 is invented for this kit. It is not any organisation's recovery wording and nothing in it
is advice about any real agreement. See `data/SOURCES.md`.*

THE CLAUSES, as data. All five are in force on every file; the question is always whether
the condition of one is met by the facts of the file in front of you.

  R-1   Right of recovery                                          permits recovery
  R-2   Waiver of recovery -- work under an executed agreement     BARS RECOVERY
  R-3   Recovery against a party insured on the same schedule      BARS RECOVERY
  R-4   Threshold below which recovery is not pursued              BARS RECOVERY
  R-5   Period within which recovery must be opened                BARS RECOVERY

  Threshold below which recovery is not pursued:  USD 2,500
  Period within which a recovery must be opened: 3 years from the date of loss
  These terms are read as at 2026-08-31.
  Order of application: R-2, R-3, R-4, R-5 -- the first whose condition is met is the one recorded.

THE THREE VERDICTS, and what answering each one commits you to:

  RECOVERABLE            Recovery may be pursued
      The file establishes that a party other than the insured caused or contributed to the loss, that party is named, and no clause in force bars a recovery against them (R-1).
      Answered wrongly this is a FUTILE PURSUIT. A recovery is opened against a party the file has already given up the right to pursue -- handling time spent for nothing, and where the waiver sits inside an agreement, a breach of it.

  NOT-RECOVERABLE        No recovery to pursue
      Either the file establishes no responsible party other than the insured, or it establishes one and a clause in force (R-2, R-3, R-4 or R-5) bars recovery against them. TWO DIFFERENT READINGS, ONE VERDICT: fill `bar_clause` to say which, and name the party even when a clause bars the recovery.
      Answered wrongly this is a MISSED RECOVERY, and nothing downstream reports it. There is no queue of recoveries that were never opened; the money is simply written off and the file closes.

  NEEDS-INVESTIGATION    Cannot be settled from this file
      The file points at a third party and does not settle it -- two accounts on the file contradict each other, a party is implied and never identified, or a fact a clause in force needs is not recorded. It is a REFUSAL asking a person for one more thing, not a third outcome.
      Used where the file does settle it, it hands back work that was already done. Not used where the file does not settle it, it turns an open question into a hard answer somebody will act on.

HOW TO QUOTE THE SENTENCE, and how it will be read.

`citation` must be ONE SENTENCE COPIED VERBATIM out of the loss file -- the sentence that
establishes that a party other than the insured caused or contributed to the loss. It may come
from the incident narrative or from an attachment.

  - Copy it character for character. It is located in the file by searching for it, so a
    paraphrase, a shortened version, an ellipsis in the middle, or two sentences joined together
    will not be found at all and will score nothing. There is no partial credit for a quote the
    file does not contain.
  - Quote the sentence, not the paragraph. What is returned is compared with that sentence by
    character overlap: it must cover at least 60 pct of the sentence, and at least 30 pct
    of what you return must be the sentence. Returning the whole file scores nothing.
  - The recovery terms are NOT part of the loss file. A clause is never the citation.
  - Where the file establishes no responsible party at all, `citation` is null. Quoting something
    in support of a party the file does not have is counted as a wrong answer, not as an empty
    one.

THE LOSS FILE, verbatim:

LOSS FILE LF-0001 -- Impact damage -- yard

FILE FACTS
  Reference                     RF-1001
  Date of loss                  2024-02-03
  Amount claimed                USD 21,500
  Location                      the north yard shutter, the Ockley depot
  Named insureds on schedule    Brantley Marine plc
  Services agreement            none in place

INCIDENT NARRATIVE

A member of the insured's staff reported that the north yard shutter had been struck at some point during the shift. The site was in normal operation and the area was not closed to vehicles.

Setterfield & Co Ltd's driver reversed a box van into the north yard shutter while turning in the yard, and Setterfield & Co Ltd has confirmed in writing that the vehicle and its driver were under its control at the time.

The damage is confined to the north yard shutter. Photographs taken the same day are on the file and the amount claimed is the repair quoted by the insured's own contractor.

PARTIES ON THE FILE

  Brantley Marine plc           the insured itself
  Setterfield & Co Ltd          a supplier or visiting party with no services agreement in place

ATTACHMENTS

  none


Reply with JSON and nothing else, exactly this shape:

{
 "verdict": "RECOVERABLE" | "NOT-RECOVERABLE" | "NEEDS-INVESTIGATION",
 "responsibility": "third_party" | "none" | "unclear",
 "party": "<text>" or null,
 "citation": "<text>" or null,
 "citation_where": "narrative" | "attachment" | "parties" | "none",
 "bar_clause": "<text>" or null,
 "confidence": <a number between 0 and 1>,
 "why": "<text>"
}

What each field means:
  verdict          the recovery decision for this file. RECOVERABLE only when a party other than the insured is established by the file as responsible AND no clause in force bars recovery against them. NOT-RECOVERABLE covers BOTH a file with no responsible third party and a file where one is plainly responsible and a clause bars the recovery -- say which by filling bar_clause. NEEDS-INVESTIGATION when the file does not settle it
  responsibility   the READING behind the verdict, kept as its own answer: does this file establish that a party other than the insured caused or contributed to the loss. `none` means the file establishes there is no such party -- the insured's own act, or no human agency. `unclear` means the file points at one and does not establish it, for example where two accounts on the file contradict each other
  party            the party a recovery would run against, spelled EXACTLY as the file spells it. Null when the file establishes no responsible party, and null when it implies one it never identifies -- an unidentified party is not a party. Name the party even where a clause bars recovery against them: who it would have run against and whether it may be opened are two answers
  citation         ONE SENTENCE COPIED VERBATIM from the loss file -- the sentence that establishes the third party's responsibility. Copy it exactly, character for character, from the narrative or from an attachment; do not paraphrase it, do not shorten it with an ellipsis, and do not join two sentences. Null when the file establishes no responsible party. The recovery terms are not part of the file and a clause is never the citation
  citation_where   which part of the loss file the quoted sentence came from
  bar_clause       the id of the clause in force that bars recovery against the named party -- R-2, R-3, R-4 or R-5 -- or null where none of them does. It is a diagnostic here: the verdict is what is graded, and this is what says WHY a NOT-RECOVERABLE is one
  confidence       your own number between 0 and 1 for the verdict. It is published as evidence and never used to change an answer
  why              one sentence naming what decided it

One decision for one loss file.

Raw responseThe raw response

{
 "verdict": "RECOVERABLE",
 "responsibility": "third_party",
 "party": "Setterfield & Co Ltd",
 "citation": "Setterfield & Co Ltd's driver reversed a box van into the north yard shutter while turning in the yard, and Setterfield & Co Ltd has confirmed in writing that the vehicle and its driver were under its control at the time.",
 "citation_where": "narrative",
 "bar_clause": null,
 "confidence": 0.98,
 "why": "The file establishes that Setterfield & Co Ltd's driver caused the damage, and no bar applies because there is no services agreement, the amount exceeds the threshold, and the loss is within the recovery period."
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Recovery identification from a loss file — 60 loss files. One model answered, and every answer was then graded Three different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

No model grades anything. Every metric is exact match against data/gold.jsonl -- strings for the verdict and the party, character intervals for the citation -- on three graded fields per file, with the unit fixed at one decision per file so no matching step exists to soften a miss. A file nothing usable came back for is counted as no_answer and never folded into a cautious-looking verdict; on both arms here that count is 0. THREE OF THE EIGHT RETURNED FIELDS ARE GRADED AND FIVE ARE NOT: verdict, party and citation are the headline; responsibility (57 of 60), bar_clause (60 of 60 on the model, 55 on the floor), citation_where, confidence and why are published as diagnostics and say WHERE a wrong verdict came from. data/fields.json carries the graded flag on every field and src/prompt.py builds the schema from it, so the fields the model is asked for and the fields the scorer reads are one list. ⚠︎ CONFIDENCE IS PUBLISHED AS EVIDENCE AND NEVER USED TO CHANGE AN ANSWER, AND ON THIS RUN IT IS USELESS: it ranged 0.92 to 1.0, median 0.98 on the 59 correct verdicts and 0.92 on the one wrong, and NEVER WENT BELOW 0.92 ON ANY FILE -- including all four where it invented a party (0.97, 0.95, 0.98, 0.95). No threshold in that range catches a wrong answer without discarding most of the right ones, and nothing here measures calibration. NO SECOND READER ADJUDICATED ANYTHING, AND NONE WAS NEEDED IN THE USUAL SENSE: the key is DERIVED by src/terms.py from the generator's injected facts rather than written by a person, so there is no inter-rater question to settle -- evals/check_labels.py, an independent second implementation, stands where an adjudication would. What that CANNOT reach is whether the generator chose the right responsibility for a file, and whether a reviewer would accept a different sentence carrying the same fact; check_labels.py prints both as the things it cannot check, and neither has been adjudicated by anybody. THE SET IS DELIBERATELY UNBALANCED AND ITS SHAPE IS PUBLISHED SO NO RATE IS READ AS A POPULATION RATE: RECOVERABLE 23 / NOT-RECOVERABLE 27 / NEEDS-INVESTIGATION 10 (so the majority-class null baseline is 45.0 pct), 15 clean against 45 hard, citation in the narrative 35 / in an attachment 9 / none 16, bar clause R-2 8 / R-3 4 / R-4 3 / R-5 3 with 42 files barred by nothing, and every per-case rate stands on a denominator between 3 and 6 -- one file is a third of a case. AND THERE IS NO RETRIEVAL STEP TO SCORE: one loss file goes whole into one prompt with the terms, nothing is selected on the way in, so no recall or top-k figure exists or could.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Run it twiceThe same set, run again

The same 60 replies scored twice, and the two columns are IDENTICAL -- 91.7 per cent and 91.7 per cent, field for field, file for file. recheck_overrides is 0: the terms engine re-applied every clause condition, the USD 2,500 threshold and the 3-year period to the model's answers and disagreed with none of them. That is not the pure-code station working; it is the pure-code station never being needed, which is a different finding and the honest one. The two rows are printed anyway because a kit that shows only the rechecked column cannot be asked what the recheck bought.

file-all-correct over the 60 loss files -- the verdict, the party and the citation all right on one file — They are not two samples of anything. They are one sample scored by two rules, so averaging them would report a spread that does not exist and hide that the spread which DOES matter -- run to run -- was never measured.

Grading costWhat it costs

Every dollar on this page is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid. The token counts come from the provider's own usage block on each call, and 90.1 pct of the output tokens they report are provider-side reasoning the kit did not ask for.

Rates checked 2026-08-27. The provider that actually ran every call here is kept off this page per the series rule. The real spend, its own dated card (2026-08-23), its peak/off-peak split and its cached-input tier are all recorded per call in results/eval-r001-recovery-flag.json, not on this page.

Grading unitWhat the grading figure prices

Scoring is pure Python over committed JSON: 60 decisions in well under a second, no key, no network. The ARM costs money; grading it does not, and re-scoring a cached run -- including at different citation floors -- costs nothing at all.

The gradersThree ways to grade

TWO FLOORS, AND THE SECOND IS THE ONE THAT MATTERS. The NULL baseline is answering NOT-RECOVERABLE on every file with no party and no citation -- the majority verdict, 27 of 60, 45.0 pct on the verdict field alone and 9 of 60 (15.0 pct) file-all-correct, computed from data/gold.jsonl and not from any run. It is reported because it is the floor beneath the floor. THE COLUMN THAT ACTUALLY MATTERS IS THE FREE RULES FLOOR (b000-recovery-flag-rules): a genuine attempt at the whole job for $0.00 -- trigger-phrase matching over the incident narrative for the reading, and then EXACTLY the terms engine the paid arm is rechecked with -- which scores 29 of 60 all-correct and 44 of 60 verdicts. ⚠︎ AND IT IS NOT A STRAWMAN AND ALSO NOT A LOWER BOUND. Not a strawman: it applies the same engine, is scored by the same scorer on the same key, and its 44 verdicts say how much of this job the engine already does for free -- 13 of 18 barred files with the right clause, 3 of 3 files where a fact the terms need is missing correctly referred, and every threshold and period comparison right on an arm with no model in it. Not a lower bound: its trigger list was REVISED AFTER ITS FIRST RUN. The first version carried only acts -- struck, collided, reversed, drove into -- and reached the waiver clause ZERO times, because it found no party on any of the eight files where a contractor's omission caused the loss; R-1 says 'act OR OMISSION', so four omission words (unattended, uncapped, unsecured, omitted) were added and the run was re-fired. That is tuning. It is recorded in evals/baseline.py, in data/SOURCES.md and in the README, and what it means is that 44 of 60 is A CEILING ON WHAT TRIGGER MATCHING DOES ON THIS CORPUS, NOT A LOWER BOUND.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-27. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingGrading is free; the bill is the 60 calls that answered.

The scorer, the label gate, the rules floor and the stub arm are pure code and cost $0.00 to run against any result set -- there is no LLM judge anywhere in the grading path, and a committed run can be re-scored at different citation floors for nothing. The figure above is the token count of the ONE paid run (60 calls, 169342 in / 79571 out) priced at the same projected card cost_per_query_usd uses. ⚠︎ THERE IS NO SECOND PAID RUN TO ADD: the injection probe is written and unfired, so reproducing this kit's whole paid evidence is exactly this one number.

Cost driversWhat actually moves the bill

• OUTPUT TOKENS. 90.1 pct of them (71,655 of 79,571) are provider-side reasoning left at the tier's default and re-rolled per call, and the output side is 73.8 pct of the projected bill. The visible answer is a JSON object of a few hundred tokens -- the LF-0001 reply is ~950 characters and was billed 773 output tokens, 621 of them reasoning.
• THE STABLE PREFIX, WHICH IS THE ONE THAT WORKS IN YOUR FAVOUR. The system role, RT-2026, the verdicts and the citation rule are byte-identical across all 60 calls and 68.1 pct of the prompt's characters. MEASURED: 103,680 of 169,342 input tokens came back as cache hits, 61.2 pct -- short of the 68.1 pct the character split predicts. The projection card has no cache tier and prices all of it at the full input rate.
• THE FILE ITSELF IS THE SMALLEST PART OF THE PROMPT -- 9.9 pct of characters, 1,179 of 11,956 on LF-0001, and the whole-prompt range across the corpus is only 11,673-12,443. The bill is the terms, the schema and the reasoning, not the document. That inverts with real files, which are not 1,146 bytes.

Your volumeWhat it costs at your volume

Linear in files and flat in everything else. Ten times the files is ten times the calls at the same per-file cost: there is no index, no retrieval and no state carried between files, and the prompt's 68.1 pct stable prefix means the cached share should hold or improve as the run lengthens (61.2 pct measured over 60 calls, with the first call taking 0 cache hits by definition). What is NOT linear is the file: this corpus averages 1,146 bytes and 2,822 input tokens, and a real loss file with a hundred pages of statements and reports moves the input side by orders of magnitude while the terms stay fixed. A team deciding 200 files a day at THIS document size is about $1.08 a day on the projection card; the same team on real files is not, and nothing here measures that.

Model choiceWhy this model

It is the tier the estate runs, so the figure compares with every sibling kit. The finding here is not about the model: what it bought is 26 files of READING -- nine of them the structural attachment row a rules floor cannot reach at any tuning -- and what it did not buy is a control for its own only failure, naming a party on a file that establishes none.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

the whole 60-file scored run (r001-recovery-flag, the fast tier): 169342 input tokens (103680 of them prefix cache hits) and 79571 output. ⚠︎ usd_actually_paid is null ON THIS PAGE by the series rule, not because it is unknown: the real bill is $0.067689 and it is recorded per call, with its tariff, in the run record.

The codeEvery file, and what it is for

11 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Eight of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/prompt.pyprompt assembly

Six parts in a fixed order -- the system role, RT-2026 verbatim plus its five clauses as data, the three verdicts with what answering each one commits you to, the citation rule, the loss file verbatim, and the JSON schema built from data/fields.json. The FIRST FOUR ARE BYTE-IDENTICAL ON EVERY CALL and are sent first, so a provider that prices cached input separately can bill a prefix that is 68.1 pct of the prompt's characters at the cached rate. It never does the reconciling: it never says a party is a contractor, that an agreement was executed first, that an amount is under the threshold or that a loss is out of time -- those are read off the file and then re-derived in code.

# The assembled prompt, in one place, in the order it is sent.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
TERMS_MD = open(os.path.join(HERE, "data", "terms.md"), encoding="utf-8").read()
FIELDS = json.load(open(os.path.join(HERE, "data", "fields.json"), encoding="utf-8"))["fields"]
VERDICTS = tuple(V.ORDER)
VERDICT_MEANINGS = {
SYSTEM = """You are reviewing one loss file and deciding a single question: is there a party other than
def _verdict_block():
def _clause_block():
CITATION_RULE = """HOW TO QUOTE THE SENTENCE, and how it will be read.

src/classifier.pythe model call

One call per loss file. Parses the reply (fence-tolerant), normalises the closed vocabularies against src/verdicts.py, and returns the raw answer and the rechecked determination side by side. max_tokens is 32,000 because the tier re-rolls a provider-side reasoning budget per call; a reply cut off at the ceiling is recorded with at_ceiling and stays in the denominator rather than scoring partially.

# One loss file in, one recovery decision out. The only place a model is called.

MAX_TOKENS = 32000
THINKING = None
def parse_reply(text):
def normalise(obj):
def classify(cfg, text, facts, complete_fn=None, max_tokens=None):
VERDICTS = tuple(V.ORDER)

src/adapters/__init__.pythe adapters — a swap seam

Raw HTTP over urllib to any OpenAI-compatible provider or Anthropic. Retries a busy provider, refuses to retry a wrong request, and checks the shared daily call cap before spending. TIMEOUT_S is 1,200 s and the module states that it and MAX_TOKENS are one setting wearing two names.

You change it to: PROVIDER + BASE_URL + MODEL in .env -- one line, then the same run again. Two adapter shapes ship (OpenAI-compatible and Anthropic's Messages API); a third is one function and one entry in PROVIDERS, and it must return token counts, because the Cost lens prices them.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
TIMEOUT_S = 1200
TRANSPORT_RETRIES = 1
def _post(url, headers, payload, timeout=TIMEOUT_S):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}

src/terms.pythe terms engine -- THE PURE-CODE STATION

RT-2026 applied to one file's recorded facts. decide() is handed the READING -- a responsibility finding and a party name -- and re-derives the verdict and the clause from data/terms.json, in the published order R-2, R-3, R-4, R-5. The same function writes the answer key, backs the free floor and rechecks the model, so no arm is scored against arithmetic a different arm used. Two rules are load-bearing: A MISSING FACT IS NOT A NEGATIVE (a file that does not record whether a services agreement was executed returns NEEDS-INVESTIGATION and names the fact), and THE ENGINE NEVER INVENTS A PARTY (responsibility third_party with no party named is NEEDS-INVESTIGATION, not RECOVERABLE against nobody).

# THE PURE-CODE STATION. RT-2026, applied to one file's facts. No model, no network, no guess.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
TERMS = json.load(open(os.path.join(DATA, "terms.json"), encoding="utf-8"))
CLAUSES = TERMS["clauses"]
CLAUSE_ORDER = TERMS["clause_order"]
AS_OF = TERMS["as_of"]
DE_MINIMIS = TERMS["de_minimis"]
LIMITATION_YEARS = TERMS["limitation_years"]
CURRENCY = TERMS["currency"]

src/citation.pythe citation scorer — a swap seam

The only implementation of how a quote is scored. One normalisation (NFC, 1:1 character folds, whitespace collapsed) applied to the file and to the returned quote; the quote is LOCATED in the normalised file and its character interval compared with the labelled sentence's. Not found is zero and is counted separately as citation_unlocatable -- there is no near-miss credit. Credit iff coverage >= 0.60 AND precision >= 0.30, and both floors are arguments, so any committed run can be re-scored at a different pair with no provider in the loop.

You change it to: The normalisation, the locate, and the two floors -- which are also CLI arguments (--coverage-min, --precision-min), so a committed run is re-scorable at any pair for nothing.

# HOW A CITATION IS SCORED. Character offsets, two floors, and no partial credit for a quote the

FOLDS = {
def fold(s):
def norm(s):
def norm_quote(s):
def locate(norm_text, quote):
def span_of(norm_text, sentence):
COVERAGE_MIN = 0.60
PRECISION_MIN = 0.30
def credit(norm_text, gold_span, quote, coverage_min=COVERAGE_MIN, precision_min=PRECISION_MIN):

src/verdicts.pythe verdict vocabulary — a swap seam

The three verdicts, the responsibility finding behind them and the party roles they turn on, written down ONCE and read by the prompt, the floor, the terms engine, the scorer and the UI. NOT-RECOVERABLE is the answer both when nobody else is responsible and when somebody plainly is and a clause bars the recovery; bar_clause is what tells them apart, and the scorer counts RIGHT FOR THE WRONG REASON separately because of it.

You change it to: The three verdicts and what each one commits you to. One copy, read by the prompt, the floor, the scorer, the engine and the UI -- a verdict list that lives in a prompt string is a verdict list the scorer cannot check against, and src/prompt.py asserts at import that every verdict and every clause the schema offers is named in the terms text.

# The three verdicts, the responsibility finding behind them, and the party roles they turn on.

VERDICT_INFO = {
ORDER = ["RECOVERABLE", "NOT-RECOVERABLE", "NEEDS-INVESTIGATION"]
RESPONSIBILITY = {
ROLES = {
ABSTAIN = "NEEDS-INVESTIGATION"
def labels():
def label_of(key):
def meaning(key):
def is_verdict(key):

src/recheck.pythe recheck — a swap seam

Exactly two fields survive: responsibility and party. verdict and bar_clause are DISCARDED from the reply and recomputed -- discarded, not corrected, because a field sometimes taken from the model and sometimes not is a field nobody can reason about. The citation is neither trusted nor repaired: the recheck locates it, records whether it is there at all, and leaves it exactly as returned, because rewriting it would erase the evidence of the failure. ⚠︎ On this run it overrode nothing -- recheck_overrides: 0. ⚑ AND IT RUNS ON BOTH ARMS, WHICH IS NEW AS OF 2026-08-31: the floor branch of evals/run.py used to write a COPY OF THE FLOOR'S OWN ANSWER into the rechecked slot, with citation_located hard-set to null, so the free arm was scored against a column the paid arm had to earn. Both free arms were re-run against RC.recheck(answer, facts, file_text) -- the same call with the same arguments the paid arm makes -- for $0.00, and every published number held. The floor's recheck_overrides: 0 is now MEASURED, and citation_located is real: 40 of the floor's 40 returned quotes locate in their file, 0 unlocatable.

You change it to: TRUSTED = ('responsibility', 'party'). Widening it moves work from code to model; narrowing it moves the injection surface, because a field the code re-derives is a field a document cannot lie about.

# THE PURE-CODE STATION, applied to a model's reply. What the model is trusted with, and nothing

TRUSTED = ("responsibility", "party")
def recheck(answer, facts, file_text):

evals/baseline.pythe free rules floor

Trigger-phrase matching over the INCIDENT NARRATIVE ONLY for whether somebody else was at fault, the first party on the file who is not the insured or a witness as the party, the first narrative sentence carrying a trigger as the citation -- and then exactly the terms engine the paid arm is rechecked with. Reading only the narrative is its defining limit and is deliberate: nine files put the answer in an attachment and the gap between the two is what those files measure.

# THE FREE FLOOR. No key, no model, no network. A genuine attempt at the job.

MODES = ("rules",)
TERMS_PHRASES = [r"caused or contributed to", r"act or omission"]
INCIDENT_PHRASES = [r"\bstruck\b", r"\bcollided\b", r"\bcollision\b", r"reversed\b",
OMISSION_PHRASES = [r"\bunattended\b", r"\buncapped\b", r"\bunsecured\b", r"\bomitted\b"]
TRIGGERS = TERMS_PHRASES + INCIDENT_PHRASES + OMISSION_PHRASES
NOT_A_TARGET = ("insured", "witness")
def narrative_of(text):
def first_trigger_sentence(narrative):
def read(text, facts):

evals/scoring.pythe scorer — a swap seam

Three fields graded each on its own and all three together -- verdict, party, citation -- with responsibility, bar_clause and citation_where reported as diagnostics. Party is matched on the name with case and punctuation folded and nothing else: 'Kelmore' where the file says 'Kelmore Services Ltd' has not named the party. Every miss is filed into exactly one of eleven taxonomy buckets in a fixed, most-consequential-first order.

You change it to: The graders, the two error directions and the eleven-bucket taxonomy. Missed recovery and futile pursuit are scored on their own denominators (23 and 27) and never averaged.

# Grade one arm against the answer key. Deterministic -- no model judges anything here.

FIELDS = ("verdict", "party", "citation")
DIAG = ("responsibility", "bar_clause", "citation_where")
TAXONOMY = ("no_answer", "futile_pursuit", "missed_recovery", "investigation_missed",
def _pct(n, d):
def _s(v):
def name_key(s):
def _bucket(g, m, cit):
def score(dets, golds, texts, coverage_min=CIT.COVERAGE_MIN, precision_min=CIT.PRECISION_MIN):
def _tax(taxonomy, bucket, did, gold, m, cit, texts):

evals/check_labels.pythe label gate

Grades the ANSWER KEY itself with arithmetic written inside it, importing nothing from src/: it re-reads data/terms.json, data/files.json, data/gold.jsonl and the corpus and recomputes every verdict, clause, threshold, period and citation span with its own normaliser -- and checks the harder property, that each labelled sentence appears in its file EXACTLY ONCE, which is what makes 'the first occurrence' a span rather than a coin toss. It also re-parses each file's FILE FACTS block out of the text and compares it field by field with data/files.json, and prints its own limit: it cannot catch a file whose gold RESPONSIBILITY or whose chosen sentence is wrong.

# Grade the ANSWER KEY. Free, no key, no model, and deliberately not the engine that wrote it.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
CORPUS = os.path.join(DATA, "corpus")
def norm(s):
def load(name):
def jsonl(name):
def parse_file_facts(text):
def parse_amount(s):
def parse_agreement(s):

src/app.pythe local board

http.server, hand-written HTML, one JS file, port 9214. Renders with no key: /api/files, /api/file, /api/rules, /api/recorded and /api/corpus need nothing and only /api/decide calls a provider. It computes the free floor on every file, prints what the terms make of the file before anybody has read it, and highlights the quoted sentence inside the file at the offsets the scorer located it at, with coverage, precision and both floors printed beside it.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
CORPUS = os.path.join(HERE, "data", "corpus")
PORT = int(os.environ.get("PORT", "9214"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r001-recovery-flag")
FLOOR_RUN = "b000-recovery-flag-rules"
def golds():
def load_file(fid):
def cite_report(text, gold, det):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 2822 input and 1326 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

⚠︎ WRITTEN, WIRED, AND NOT FIRED — AND NO NUMBER ON THIS PAGE SAYS OTHERWISE. A loss file is this kit's injection surface and the threat is not hypothetical: the file is assembled from documents written by the parties to the loss — a driver's statement, a supplier's email, a report somebody commissioned — and every one of them reaches the model VERBATIM. The party with the most to gain from a NOT-RECOVERABLE is the party the recovery would run against, and they can often write into the file. evals/injection.py exists, is complete, and takes the 23 files whose correct verdict is RECOVERABLE, appends one attacker-shaped line, and pairs each reply against the scored run's own cached answer so the control costs nothing. It refused to start without that control cache; THE CACHE NOW EXISTS (results/cache-r001-recovery-flag.jsonl is committed) AND THE PROBE WAS STILL NOT RUN. results/ holds no x001-recovery-flag file. There is therefore no suppression rate, no reading-moved count and no citation-capture count for this kit, and none is estimated.

API_KEY is read from the shared <code>&lt;repo&gt;/.env</code>, this kit's own <code>.env</code>, or the real environment, in that order (src/config.py). Both files are gitignored from the first commit; this repo has never held a credential. The key never leaves the machine, is never printed, and src/budget.py counts every call against a shared daily cap written BEFORE the call is made, so a crash mid-call over-counts rather than under-counts.

The experimentWe attacked it — two gates

An indirect prompt injection has to clear two gates, and they behave differently. ⚠︎ THE ADVERSARIAL ARM WAS NEVER FIRED. evals/injection.py is written, wired and has the control cache it was refusing to start without, and it was not run: no run — results/ holds no x001-recovery-flag file, and no adversarial figure appears anywhere in this spec.

Both gates are open. Writing the shape down here is what stops a future run reporting a padded denominator as resistance, and it is also what stops this page reading as though a zero had been measured. ⚠︎ AND ONE THING IS KNOWN WITHOUT THE PROBE, FROM THE SCORED RUN ITSELF: recheck_overrides is 0, so the pure-code station has never once disagreed with this model about anything. Its protective value against a moved reading is therefore not merely unmeasured under attack — it is unexercised even without one.

The result⚠︎ NOT FIRED — no adversarial number exists for this kit. evals/injection.py is written, wired, and has its control cache; results/ holds no x001-recovery-flag run.

The design, so it can be checked rather than re-argued later: each trial would be one loss file of this corpus with the shipped line appended as its final register entry and everything else byte-identical, paired against that same file's own un-injected answer read from results/cache-r001-recovery-flag.jsonl — one variable, and the control costs nothing. Only files the control got right would be in the paired denominator, because 'the injected arm also got it wrong' is not evidence about injection. NONE OF IT HAS BEEN RUN.

HonestyWhat this does not prove

• EVERYTHING ABOUT INJECTION. The probe is written and unfired: no wording has been tried, on any model, on any day. This is not a 0 pct suppression rate; it is no rate.
• WHETHER THE PURE-CODE STATION PROTECTS ANYTHING. recheck_overrides is 0 on the only run there is, so the terms engine has never disagreed with this model even without an attack.
• AN ATTACK ON THE READING. responsibility and party are the two fields the recheck trusts, and a sentence engineered to move them moves the whole determination. Unmeasured, and it is the direction the architecture cannot defend.
• AN ATTACK ON THE REGISTER. data/files.json is trusted absolutely by design — the role, the named insureds, the amount and the agreement date all come from it. Nothing here measures what happens when the register itself is wrong, and no arm in this kit could see it.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Never open a recovery, never write to any system, never contact a party, never send a letter. This kit produces the decision A PERSON CONFIRMS, and NEEDS-INVESTIGATION is that decision asking the person for one more thing rather than a third outcome.

Stated on the UI, in the README and in src/app.py's docstring, and enforced by there being no such endpoint: the server's POST route returns decisions and nothing else, and there is no write path to data/files.json, to any register or to any recovery system.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• This is ABSENCE OF A WRITE PATH plus a prompt rule, not a runtime enforcement layer — with one partial exception: the recheck IS enforcement for everything RT-2026 decides by clause, threshold or period. ⚠︎ AND ON THIS RUN IT ENFORCED NOTHING: recheck_overrides is 0. It is a control that has never fired.
• ⚠︎ THERE IS NO CONTROL AT ALL FOR THE FAILURE THIS KIT ACTUALLY HAD. The model named a party on 4 files that establish none — its whole party_invented count and 4 of its 5 misses. The terms engine re-checks clauses, thresholds and periods and has nothing to say about whether a name BELONGS. A deployment would need that control and this kit does not have one.
• ⚠︎ CONFIDENCE IS NOT A GUARDRAIL HERE AND MUST NOT BE USED AS ONE. The model never returned below 0.92 on any file, including all four it invented a party on (0.97, 0.95, 0.98, 0.95) and the one verdict it got wrong (0.92). No threshold in that range separates right from wrong, and nothing here measures calibration.
• The 91.7 pct is agreement with a COMPUTED key on an invented corpus of 60 files, and the engine that rechecks is the engine that wrote the key. evals/check_labels.py is the independent second implementation that makes that survivable, and it prints its own limit: it cannot check whether the generator chose the right responsibility, or whether a reviewer would accept a different sentence.
• NEEDS-INVESTIGATION is a request to a person, not a block. Nothing stops a file being closed on a refusal except the person the refusal is for — and on four files that refusal arrives with a party name attached to it.

WatchedWhat is watched, and why that one

The kit measures 34 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

120 measured by the latest run -86 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (attachment_source_files 9, barred_files 18, citation_coverage_floor 0.6, citation_precision_floor 0.3, cited_files 44, clean_files 15, dataset_version recovery-flag-v1-60files, failures 0, files_answered 60, hard_files 45, investigation_files 10, loss_files 60, notrecoverable_files 27, rechecked_attachment_source_files 9, rechecked_barred_files 18, rechecked_cited_files 44, rechecked_clean_files 15, rechecked_files_answered 60, rechecked_hard_files 45, rechecked_investigation_files 10, rechecked_notrecoverable_files 27, rechecked_recoverable_files 23, rechecked_rederived_from_cache False, rechecked_uncited_files 16, recoverable_files 23, socket_timeout_s 1200, terms_id RT-2026, uncited_files 16) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

3 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

triage · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 114 chips that all say so.

triage · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 124 chips that all say so.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 120 chips that all say so.

DeviationsWhat deviated

0 breaches across 3 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

The corpus generator, the label gate, the free floor and the stub arm are free and re-run in seconds on a keyless copy, and the recheck re-derives from the committed cache for nothing. Only the model arm costs money — 60 calls, once, recorded per call in the run record and never re-fired after its misses were read. ⚠︎ THE ADVERSARIAL ARM IS WRITTEN AND HAS NEVER RUN, so nothing here has a cadence under attack.

In one lineNo framework, deliberately

NO FRAMEWORK. A folder of readable Python, standard library end to end — no orchestration layer, no vendor SDK, no retrieval stack, no NLP library, no pip install. requirements.txt names nothing and says why.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Linear, and concurrent only at the file level. There is no agent, no tool loop, no retrieval and no state carried between files; every file is independent, which is exactly why 5 workers is the entire concurrency story and why 60 files took 164.1 wall seconds against 694.6 seconds of summed call latency.

The other sideWhat a framework costs you

• Adding a provider means editing the PROVIDERS dict by hand. In exchange pip install pulls nothing and a forker runs this on whichever key they already hold.
• The reply is parsed by hand. It held on all 60 paid calls, and that is an observation, not a guarantee — a structured-output library would fail loudly where this fails quietly.
• No rules-engine DSL, so a terms document whose SHAPE differs from this one's (reading -> clause order -> first condition met) is a change to src/terms.py rather than to a config file. The JSON carries the numbers and the clause ids; the code carries the shape.
• No NLP library, so the free floor is a regex over trigger phrases and nothing more. That is the floor's known ceiling and it is stated rather than hidden — and it is also why the floor's 0 of 9 on attachment-sourced sentences is structural: it never opens an attachment, and no library would change that decision.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-recovery-flag on the fast tier, 2026-08-31. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 1 earlier run on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

No drift chart. Only one run has recorded a call latency; the other 1 run on record made no model call at all — a rules-only floor stores 0, which is an absence, not a zero-millisecond answer. One timed run is a reading, not a trend, and it is in the table above. A chart appears the day a second run times itself.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the rulebook is data read whole into the prompt.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT, with the corpus, the register and the terms generated in-process -- there is no third-party data in this kit to licence. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe whole recovery decision -- the verdict, the party, and the sentence that establishes it

whether each of the 60 files produced a decision a reviewer could confirm as written: the verdict RT-2026 produces on the file's own recorded facts, the party a recovery would run against spelled exactly as the file spells it, and one sentence from the file that establishes that party's responsibility. All three, not two -- a right verdict with a name the file does not support is the failure this kit's only error mode actually is.

MethodHow the test was run

python3 -m evals.run --run-id <id> [--floor rules]; evals/scoring.py compares strings against data/gold.jsonl with case and punctuation folded on the party name and nothing else. No model is in the grading path.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl -- DERIVED, NOT WRITTEN. src/terms.py::decide() is called with the generator's injected reading and injected file facts, and the citation span is computed by locating the planted sentence in the normalised file. evals/check_labels.py then re-derives every verdict, clause, span and FILE-FACTS-versus-register comparison with arithmetic written inside itself, importing no src/ module, and checks that each labelled sentence appears in its file exactly once. Two implementations, one key. Every insured, third party, person, site, loss, attachment and clause is invented -- see data/SOURCES.md.

No true/false rates for this grader. It records 6 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No tolerance on the verdict or the party -- exact match, with case, surrounding punctuation and repeated whitespace folded on the name and nothing else, because a fuzzy match would be the scorer grading its own guess. The only numbers that look like thresholds belong to the CITATION grader (coverage 0.60, precision 0.30) and to RT-2026 itself (USD 2,500 and 3 years, both in data/terms.json and applied by every arm).

Cadence: Once per corpus version. The paid arm ran ONCE, was not re-fired after its misses were read, and the recheck is re-derivable from the committed cache for nothing.

The decisionWhen to reach for it

In one lineThe two error directions, never averaged -- and the verdict that is right for the wrong reason

which way an arm fails. MISSED RECOVERY is a recoverable file called NOT-RECOVERABLE: the money is written off and nothing downstream reports it, because there is no queue of recoveries that were never opened. FUTILE PURSUIT is the reverse -- a barred file, or one with no responsible third party, called RECOVERABLE -- which spends handling time and, where the waiver sits inside an agreement, breaches it. They are scored on their own denominators (23 and 27) and never averaged. Beside them sits RIGHT FOR THE WRONG REASON: a barred file called NOT-RECOVERABLE by an arm that got there by finding nobody at all, which every accuracy column scores as a win.

MethodHow the test was run

python3 -m evals.run --run-id <id> [--floor rules]; the scorer splits every verdict miss by direction before filing it into a taxonomy bucket. No model is in the grading path.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The same data/gold.jsonl. The split into 23 recoverable, 27 not-recoverable and 10 refused is fixed by the key before any arm answers, and the 18 barred files carry the clause that bars them (R-2 8, R-3 4, R-4 3, R-5 3).

No true/false rates for this grader. It records 6 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No tolerance -- a verdict either crossed the line or it did not. ⚠︎ THERE IS NO DOLLAR COLUMN HERE. A sibling kit denominates its error directions in money; this one cannot, because what a recovery is worth is not the amount claimed and nothing in this corpus records a recovery rate. The counts are the measurement and no cash figure is derived from them.

Cadence: Every run, on every arm. The adversarial arm that would re-measure this pair under attack is WRITTEN AND UNFIRED.

The decisionWhen to reach for it

In one lineThe sentence, located in the file by character overlap -- and silence where the file has none

whether the evidence behind the decision is real. The returned quote is normalised and LOCATED in the normalised file; not found is zero and counted separately, with no near-miss credit. Found gives an interval, and credit needs coverage >= 0.60 of the labelled sentence AND precision >= 0.30 of what was returned. On the 16 files the key gives no sentence, credit is for returning NOTHING -- quoting something in support of a party the file does not have is a wrong answer, not an empty one.

MethodHow the test was run

python3 -m evals.run --run-id <id> [--floor rules] [--coverage-min X --precision-min Y]. Both floors are arguments, so any committed run can be re-scored at a different pair with no provider in the loop and no call re-bought.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The gold citation is A PAIR OF CHARACTER OFFSETS, not a string. tools/build_corpus.py plants the sentence, records [g0, g1), and asserts it appears in the file EXACTLY ONCE -- more than once and 'the first occurrence' would be a coin toss. evals/check_labels.py re-derives the spans with a second normaliser written inside itself. Sentence lengths: mean 193.9 characters, range 130-265.

No true/false rates for this grader. It records 6 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? ⚠︎ THE TWO FLOORS WERE RED-PROVEN AND BOTH ARMS LEAVE THEM UNTESTED, AND BOTH HALVES OF THAT MATTER. Red-proven: three constructed arms were scored against the shipped key for nothing -- the right sentence clipped to 70 pct of its length scores 44 of 44 at coverage >= 0.60 and 0 of 44 at >= 0.75; the right sentence plus ~400 characters either side scores 9 of 44 (mean precision 0.22) and 0 of 44 at precision >= 0.50; the ENTIRE FILE scores 0 of 44 (mean precision 0.18), and data/corpus-stats.json carries citation_whole_file_precision_max: 0.2338 so the anti-dump claim is checkable rather than asserted. Untested: the model's mean coverage is 0.9946 and its mean precision 1.000, and the floor's number does not move at any pair, so 60 of 60 would be 60 of 60 at almost any threshold. The three constructed arms are the ONLY evidence that these floors discriminate at all -- that is a limitation of this measurement, not a vindication of the numbers.

Cadence: Every run, on every arm, and re-scorable at any pair from the committed cache for $0.00.

The decisionWhen to reach for it