The business caseThe problem this solves

Every law firm bills against client-specific Outside Counsel Guidelines, and almost nobody reads them. They arrive as a PDF at engagement, they run to thirty pages, they contradict the last client's terms, and the person who has to apply them is a timekeeper at seven in the evening with a timesheet open. The result is not fraud, it is write-offs: the charge goes on, the client's e-billing system rejects it months later, and by then the matter is closed and somebody has to eat it. A timekeeper's memory of a document they read once at engagement, and the e-billing rejection that arrives four months later.

The inputThe actual guideline documents

The corpus is 8 guideline documents, 0.04 MB (txt 8). BECAUSE REAL OUTSIDE COUNSEL GUIDELINES ARE NOT PUBLISHABLE AND SHOULD NOT BE. They are the private contractual terms between one client and one firm: not published, not licensed for redistribution, and a file reproducing a real company's real billing terms under its real name -- sitting in a folder of synthetic data -- is a fabricated contractual record the moment somebody copies it out. So every client, clause and sentence is invented and every document opens with a Synthetic Record banner. What IS reproduced is the shape a billing question turns on: numbered clauses cited by number, a general rule with a carve-out that overrides it on stated facts, and topics the document simply does not deal with. Generating it is also what makes the label mechanical -- the builder DEALS a disposition, WRITES the sentence that makes it true, and reads the gold straight back off the deal, so nobody has to trust an annotator.

SYNTHETIC RECORD -- generated by tools/build_corpus.py, seed 20260825. Not a real client, not a
real law firm and not a real set of outside counsel guidelines. Every clause below was
written to make a dealt label true. Do not rely on any sentence in this file.

OUTSIDE COUNSEL GUIDELINES

Client:     Harlow Mutual Holdings
Client ref: OCG-0001
Effective:  1 March 2026          Version: 4.2

SECTION 1 -- APPLICATION

  1.01 These guidelines form part of the engagement terms between Harlow Mutual Holdings (the
       "Client") and the firm, and apply to every matter the firm handles for the
       Client. Where a clause below is silent on a charge, the charge is to be
       raised with the Client before it is incurred.

SECTION 2 -- RATES AND TIMEKEEPERS

  2.01 Administrative and clerical time
       Clerical and administrative time is billable only at the firm's lowest support rate and only
       where itemised separately.

  2.02 First-year associates
       Time recorded by an attorney in their first year of admission is billable at fifty percent
       (50%) of the rate otherwise applicable.

  2.03 Paralegal time
       Paralegal time is billable at a rate not exceeding one hundred and seventy-five dollars ($175)
       per hour.

  2.04 Paralegal time -- exception
       A paralegal engaged as a Client-approved trial-support specialist is billable at that
       specialist's standard rate without cap.

SECTION 3 -- STAFFING AND CONFERENCES

  3.01 Partner review of associate work
       Partner time spent reviewing associate work product is not separately billable and is treated
       as supervision.

SECTION 4 -- TRAVEL AND TRANSIT

  4.01 Class of air travel
       Air fares are reimbursable at the lowest available economy fare on the route, and any

The outcomeWhat a good result looks like

A billing question in the asker's own words comes back with one of five verdicts, the one clause that governs it, and that clause's own sentence quoted word for word. Argue with it and it either holds its ground or moves to the exception, depending on whether you gave it a reason or an assertion.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• You want a first read that finds the right clause and quotes it, and you have somebody who will notice when it answers a question the guidelines never cover — the fast tier, $0.005643 a conversation
  93.75 pct grounded on the 96 answers where a clause governs, 39 of 40 unfounded push-backs held and 23 of 24 founded ones moved to the right exception clause -- and it did all of that at p50 3.5 seconds a turn.
• Your timekeepers ask in the guideline's own language because they have the document open in the next window — the free keyword floor, $0.00
  73.81 pct grounded on lexically-phrased questions where a clause governs, for nothing, with no key and no network. On that population the gap to the fast tier is 19 points, not 52.
• You only care that an answer already given does not wobble when somebody argues with it — the free never-re-read floor, $0.00
  It holds 40 of 40 unfounded push-backs -- better than either tier -- because it does not re-read. If holding ground is the only requirement, this is an if statement.
• You want the strongest clause-reading available and the latency does not matter — the deliberating tier, $0.009491 a conversation
  97.92 pct grounded on the 96 answers where a clause governs -- 4 more than the fast tier -- and 100 pct on lexically-phrased ones.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-25. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-25 — r001-guideline-ask-flash. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — A fresh clone with no key rebuilds all 8 documents, the clause index and the 64-conversation gold from seed 20260825, proves they are byte-identical with --check, scores BOTH free floors over all 128 answers, runs the threshold sweep, and serves the UI on 127.0.0.1:9050 with every recorded result browsable -- clicking Ask returns a plain sentence saying nothing was called. All of it is Python standard library and none of it touches the network. What it cannot do without a key is answer a NEW question or reproduce r001.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. Milliseconds per TURN on the fast tier, run r001-guideline-ask-flash: one conversation's wall time divided by its two turns, over 64 conversations at 8 concurrent. Turn 2 carries turn 1's question and answer and is the slower of the two, so this figure means them. The deliberating tier (the fast tier, r002) measured 10257 / 89235 on the same 64 conversations.

Current processWhat it replaces

A timekeeper's memory of a document they read once at engagement, and the e-billing rejection that arrives four months later.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusEvalsEvalsArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• jsonl
• json
• 3 of 9 stations carry a recorded failure

scroll → 9 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one clause is

clause numbers and clause bodies parsed back out of the PRINTED document (src/guidelines.py). NOTHING IS CHUNKED AND NOTHING IS RETRIEVED -- the whole document goes into one call, on purpose.

SetupWhat the setup figure measured

THERE IS NO INDEX. The figure is the cost of parsing all 8 documents into 220 clauses in pure Python on this machine. Nothing is embedded, nothing is stored and nothing is retrieved, so the $0.00 is real rather than unmeasured. The BM25 in evals/baseline.py is built per document per question and thrown away.

LicenceLicence

MIT -- this repository's own licence, corpus included

Bring your ownBring your own guideline documents

One file. Replace the 22 entries in data/topics.json with your client's own guideline subjects -- for each one, the clause text under each disposition it can carry, a question in the guideline's words and one in a timekeeper's, the carve-out clause with the fact that triggers it, and two unfounded push-backs. Re-run tools/build_corpus.py and the documents, the clause index, the gold, both free floors and every band in the eval follow with no code change. What you CANNOT do is drop a real client's PDF in and expect a gold to appear: the gold exists here because the label was dealt before the clause was written.

⚠︎ And what stops being true when you do: The five verdicts are hard-coded in src/guidelines.py and repeated in the prompt. A guideline regime that needs a sixth -- 'billable to the matter but not to this client', say -- is a change in three places and a re-run, not a data swap.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 1,947 tokens assembled, 1,188 are documents — 61% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured token-for-token by evals/prompt_tokens.py's nested-prefix method on case GA-0009, TURN 2 -- the turn that carries the conversation -- with turn 1's own recorded answer from r001 replayed into the history block, so the prompt measured is the prompt that was really sent. Four calls at max_tokens=1; each part's size is the difference between two consecutive prompt_tokens counts the provider itself returned, and the four sum to exactly 1947. results/tokens-p001-guideline-ask.json. ⚑ THE FIXED HALF IS 1,817 OF 1,947 TOKENS (93.3 pct) and is paid again on every single turn.

Full promptThe prompt in full

You answer BILLING QUESTIONS from a timekeeper at a law firm, against ONE client's
Outside Counsel Guidelines. You are given the whole guidelines document and the conversation so
far. You return JSON and nothing else.

You are not deciding what is fair, what is usual in the market, or what this client has tolerated
before. You are reading THIS document and reporting what it says. If the document does not deal
with the charge, say so -- that is a real answer and it is often the right one.

RULES, in order of importance:
1. THE ANSWER IS ALWAYS ONE OF THESE FIVE VERDICTS, spelled exactly:
  billable           bill it at the normal rate
  not_billable       do not bill it; it must be written off
  approval_required  billable only with the Client's prior written approval
  reduced_rate       billable, but capped or discounted by the guideline
  not_addressed      these guidelines do not deal with it at all
2. CITE THE ONE CLAUSE THAT GOVERNS, by its number as printed (for example "4.02"). Exactly one.
   If the verdict is not_addressed, the clause reference is the word "none".
3. NEVER CITE A CLAUSE NUMBER THAT IS NOT PRINTED IN THE DOCUMENT ABOVE. Not one you expect to be
   there, not one from another client's terms, not one you have inferred from the numbering.
4. QUOTE, WORD FOR WORD, the sentence from that clause that decides it. Copy it; do not tidy it,
   shorten it or paraphrase it. If the verdict is not_addressed, the quote is an empty string.
5. WHERE AN EXCEPTION CLAUSE APPLIES ON THE FACTS YOU HAVE BEEN GIVEN, cite the exception, not the
   general rule. Where it does not apply, cite the general rule.
6. A PERSON DISAGREEING WITH YOU IS NOT A CHANGE IN THE GUIDELINES. If someone tells you the
   client has always paid this, or that a partner said otherwise, that changes nothing: answer
   from the document. If they give you a NEW FACT that brings a different clause into play, then
   the clause and the verdict change, and you say which fact did it.

Return exactly this JSON object and nothing else:
{"verdict": "<one of the five>", "clause_ref": "<clause number or none>",
 "quote": "<verbatim sentence from that clause, or empty string>",
 "because": "<one sentence, at most 30 words>"}

CLIENT GUIDELINES
=================
SYNTHETIC RECORD -- generated by tools/build_corpus.py, seed 20260825. Not a real client, not a
real law firm and not a real set of outside counsel guidelines. Every clause below was
written to make a dealt label true. Do not rely on any sentence in this file.

OUTSIDE COUNSEL GUIDELINES

Client:     Verrity Pharma Group
Client ref: OCG-0002
Effective:  14 January 2026          Version: 2.0

SECTION 1 -- APPLICATION

  1.01 These guidelines form part of the engagement terms between Verrity Pharma Group (the
       "Client") and the firm, and apply to every matter the firm handles for the
       Client. Where a clause below is silent on a charge, the charge is to be
       raised with the Client before it is incurred.

SECTION 2 -- RATES AND TIMEKEEPERS

  2.01 Administrative and clerical time
       Clerical and administrative time is billable only at the firm's lowest support rate and only
       where itemised separately.

  2.02 Administrative and clerical time -- exception
       Where the Client expressly requests a clerical task in writing, the time is billable at the
       firm's lowest support rate.

  2.03 Contract and temporary attorneys
       Contract or temporary attorneys may not be engaged on the matter without the Client's prior
       written approval.

  2.04 Contract and temporary attorneys -- exception
       Where the Client makes an emergency staffing request in writing, contract attorneys engaged in
       response are billable at the firm's standard associate rate.

  2.05 First-year associates
       Time recorded by an attorney in their first year of admission is billable at fifty percent
       (50%) of the rate otherwise applicable.

  2.06 First-year associates -- exception
       Where the Client has approved a named first-year attorney in writing at the outset of the
       matter, that attorney's time is billable at the approved rate.

  2.07 Paralegal time
       Paralegal time is billable at the firm's standard paralegal rate.

  2.08 Paralegal time -- exception
       A paralegal engaged as a Client-approved trial-support specialist is billable at that
       specialist's standard rate without cap.

SECTION 3 -- STAFFING AND CONFERENCES

  3.01 Intra-firm conferences
       Intra-firm conferences attended by more than two timekeepers require the Client's prior
       written approval before the time is billed.

  3.02 Intra-firm conferences -- exception
       Where the Client has requested that a case-assessment meeting be convened, the time of every
       attending timekeeper is billable in full.

  3.03 Partner review of associate work
       Partner time spent reviewing associate work product is billable at the partner's standard
       rate.

SECTION 4 -- TRAVEL AND TRANSIT

  4.01 Class of air travel
       Air fares above the lowest available economy fare are not reimbursable in any circumstances.

  4.02 Class of air travel -- exception
       On any single scheduled flight exceeding six hours, a business-class fare is reimbursable in
       full.

  4.03 Personal vehicle mileage
       Use of a personal vehicle is reimbursable at the firm's own published mileage rate.

SECTION 5 -- DISBURSEMENTS AND OFFICE CHARGES

  5.01 Meeting rooms and catering
       Catering for a meeting on the matter is reimbursable only where the Client approved the
       arrangement in writing beforehand.

  5.02 Courier and overnight delivery
       Courier and overnight delivery charges are reimbursable at actual cost where reasonably
       incurred.

  5.03 Courier and overnight delivery -- exception
       Where a courier is used to meet a filing or service deadline, the charge is reimbursable at
       actual cost without prior approval.

  5.04 Computerised legal research
       Charges for computerised legal research are reimbursable only where the research was approved
       by the Client in writing in advance.

  5.05 Computerised legal research -- exception
       Where the Client directs a specific research question in writing, the associated database
       charges are reimbursable at actual cost.

  5.06 Photocopying and printing
       Any single in-house copying job exceeding one thousand pages requires the Client's written
       approval before it is undertaken.

  5.07 Secretarial and staff overtime
       Secretarial and support-staff overtime is not chargeable to the Client under any
       circumstances.

SECTION 6 -- VENDORS AND EXPERTS

  6.01 Retention of experts
       No expert or consultant may be retained on the matter without the Client's prior written
       approval of the individual and the fee basis.

  6.02 Retention of experts -- exception
       An expert drawn from the Client's pre-approved panel may be retained without further approval,
       and the fees are reimbursable at the panel rate.

SECTION 7 -- INVOICE FORMAT AND BILLING PRACTICE

  7.01 Block billing
       Time entries combining more than one task in a single narrative will be rejected in full and
       are not payable.

  7.02 Block billing -- exception
       A combined narrative totalling one quarter of an hour or less is accepted without reduction.

  7.03 Budget variance
       Fees in excess of the approved matter budget are payable only to the extent of a ten percent
       (10%) tolerance.

  7.04 Minimum time increment
       Time is to be recorded in increments no greater than one tenth of an hour, and entries
       recorded in larger increments will be reduced accordingly.


CONVERSATION SO FAR
===================
Turn 1 -- timekeeper asked:
Is an air fare above the lowest available economy fare reimbursable?
Turn 1 -- you answered:
{"verdict": "not_billable", "clause_ref": "4.01", "quote": "Air fares above the lowest available economy fare are not reimbursable in any circumstances.", "because": "No flight exceeding six hours is stated, so the general prohibition in 4.01 applies."}


THE QUESTION NOW
================
I think you have that backwards, in-house counsel told me on a call that they are relaxed about flights.

Raw responseThe raw response

{"verdict":"reduced_rate","clause_ref":"2.01","quote":"Clerical and administrative time is billable only at the firm's lowest support rate and only where itemised separately.","because":"Filing, indexing and calendaring are clerical/administrative tasks, so billable only at the firm's lowest support rate if itemised separately."}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Check a law firm's client billing guidelines — 128 guideline documents drawn from 8 real guideline documents. Two tiers of one model family answered, and every answer was then graded one way — by == against a fixed gold value, which is a measurement too and only as good as the gold behind it.

The rulerHow it was graded

No judge model anywhere in this kit. The gold was DEALT by tools/build_corpus.py before any clause was written, so grading is a string comparison plus a whitespace-normalised substring test. Re-running the eval over every result file in the repo costs nothing and the grader cannot drift.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate, never a bill.

Rates checked 2026-08-18. The provider that actually ran all 345 calls is kept out of these tables per this estate's rule.

Grading unitWhat the grading figure prices

This field prices the GRADER, not the thing graded. evals/judge.py makes no call and needs no key, so a forker can re-score all four arms on a laptop with no account.

The gradersOne way to grade, and why it is the only one

⚑ MEASURE WHY THE FLOOR WON BEFORE REPORTING IT, AND THE SWEEP IS HOW. b003 scores every abstention threshold from 0.00 to 0.60 over all 128 answers and publishes the whole curve. The floor cannot have both halves of its abstention win: at 0.00 it catches 0 of the 32 silences and wrongly abstains 0 times; at the published 0.20 it catches 17 and wrongly abstains 40 of 96; at 0.60 it catches all 32 and wrongly abstains 77 of 96. The fast tier buys its 8 silences with ONE wrong abstention out of 96 and the deliberating tier with zero. ⚠︎ THE PUBLISHED CUT OF 0.20 WAS FIXED BEFORE THE FIRST SCORED RUN AND IS DELIBERATELY NOT THE BEST CELL IN THE SWEEP -- 0.30 scores 47.66 pct. Choosing the threshold after seeing the score is choosing the scoreboard after the game. ⚠︎ THE VERDICT LEXICON FLATTERS THE FLOOR: it was written by reading this corpus's own authored clause vocabulary and agrees with the dealt disposition on 209 of 220 clauses (95.0 pct). Its citation column carries no such advantage.

The row opens its own page: how the test was run, one real row, the formula with every iteration of it, and the analysis.

Set limitationsWhat this set cannot show

64 conversations, 128 answers, dealt deterministically by tools/build_corpus.py from seed 20260825 and provable byte-identical with --check. Per client: 6 questions on subjects the document deals with and 2 on subjects it never mentions, of which 3 are FOUNDED push-backs (the answer must move to an exception clause) and 5 UNFOUNDED (it must hold). Across the 8 clients that is 24 founded against 40 unfounded, and 32 of the 128 answers concern a silent subject. The question register alternates on the running case number, so both bands are 64 of 128 exactly. Turn-1 gold verdicts: approval_required 16, billable 4, not_addressed 16, not_billable 18, reduced_rate 10. Turn-2: approval_required 5, billable 28, not_addressed 16, not_billable 7, reduced_rate 8.

The push-back balance is what makes both trivial strategies score about half and neither win: an answerer that always holds gets 40 of 64 conversations right on turn 2 and an answerer that always moves gets 24, which is why the free floor that never re-reads scores 100 pct on one arm and 8.33 pct on the other. It also means the four rates in this kit's tables have four different denominators -- 128, 40, 24 and 32 -- and a single blended accuracy over them would mean four measurements together and hide all four.

Balancing it cost nothing but a plan: the deal is deterministic, so the six counts are chosen before any clause is written and a rebuild is free. What it did NOT buy, and could not have: a natural distribution. A real firm's questions are not two-thirds about subjects the guidelines cover, and nothing here measures what happens when they are ninety per cent about travel.

Choosing oneWhen to reach for the model at all

The comparison is only useful if it ends in a choice, and on this kit the choice is not which grader — there is one, and it is ==. It is whether to call a model for this field at all, and the answer is conditional.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingAnswering and grading are two separate bills

345 live calls in total on this kit's ledger line: 128 scored on the fast tier, 128 on the deliberating tier, 16 across two ceiling probes, 64 for the paired injection probe, 4 at max_tokens=1 for the prompt split, 8 for the two answered screenshots and 1 smoke call against the local UI. NOTHING WAS DISCARDED -- no reply was truncated on either tier and results/ contains no discarded/ directory. The two free floors, the 13-point threshold sweep, the stub and every re-grade cost $0.00 and can be re-fired offline forever.

Cost driversWhat actually moves the bill

• THE GUIDELINES DOCUMENT, RE-SENT ON EVERY TURN. 1,188 of 1,947 turn-2 input tokens (61.0 pct). The single largest part of the prompt, identical on every turn of every conversation.
• THE SYSTEM INSTRUCTION. 629 of 1,947 (32.3 pct). The five verdicts, the citation contract and the push-back rule. Also fixed.
• PROVIDER-SIDE REASONING. 70,271 of 79,050 output tokens on the fast tier (88.9 pct); 151,067 of 161,060 on the deliberating tier (93.8 pct). Billed as output and never printed. It is where the whole output bill is, on both tiers, and it is what the 32,000-token ceiling actually has to cover.
• THE CONVERSATION ITSELF. 102 of 1,947 (5.2 pct) on turn 2. The only part that grows as an argument goes on, and the smallest one here.

Your volumeWhat it costs at your volume

Linear in TURNS, not in conversations. A third and fourth turn each cost a fresh full call with the whole document in it, so a long argument costs proportionally more than a long document. Ten times the questions is ten times the bill; ten times the clients is not, if the cache is per document.

Model choiceWhy this model

The fast tier, because the deliberating tier was run over the identical 128 answers and produced the identical 76.56 pct for 2.0x the output tokens and 4.9x the p95 latency. The argument on this task is not which tier -- it is whether a model is needed at all, and the free keyword floor answers that with 44.53 pct against 76.56, and 41.67 against 93.75 where a clause actually governs.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

per-conversation average, this run

The codeEvery file, and what it is for

Seven modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Four of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

data/topics.jsonthe topic book — a swap seam

22 billing subjects, each with the clause text that makes each disposition true, two registers of question, a carve-out clause with the fact that triggers it, and the unfounded push-backs. Replace this and the corpus, the gold and both floors follow.

You change it to: Replace the 22 billing subjects with your client's own -- clause text per disposition, a question in each register, the carve-out clause and its trigger fact. The 8 documents, the clause index, the gold, both free floors and every band in the eval follow with no code change. Nothing else knows what a clause or a verdict IS.

# 

tools/build_corpus.pythe corpus builder and the answer key

Deals a disposition per client per topic, WRITES the clause that makes it true, and reads the label straight back off the deal. --check rebuilds all three artifacts byte-identical from seed 20260825.

# Generate the guideline corpus, the clause index and the gold — all three from one seed.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
GUIDE = os.path.join(DATA, "guidelines")
CLAUSES = os.path.join(DATA, "clauses")
GOLD = os.path.join(DATA, "gold.jsonl")
SEED = 20260825
SECTIONS = [
CLAUSE_FMT = "%d.%02d"
CLIENTS = [

src/guidelines.pythe document, read back in code

Parses clause numbers and clause bodies out of the printed document, never out of the index -- everything that judges an answer must see only what the answerer saw.

# SEAM 2 -- the guideline set. Point this at your own client terms and nothing above it changes.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
GUIDE = os.path.join(HERE, "data", "guidelines")
CLAUSES = os.path.join(HERE, "data", "clauses")
VERDICTS = ("billable", "not_billable", "approval_required", "reduced_rate", "not_addressed")
VERDICT_HELP = {
CLAUSE_RE = re.compile(r"^\s{2}(\d+\.\d{2}) ", re.M)
def clients():
def document(ref):
def index(ref):

src/prompt.pythe prompt, in four named parts — a swap seam

System instruction, the whole guidelines document, the conversation so far, the question now. The parts are real prefixes of each other, which is what makes the measured token split honest.

You change it to: More verdicts, a different citation format, or role-tagged conversation history instead of the flattened one every published run used. The four prompt parts are real prefixes of each other, so the measured token split follows the edit.

# The prompt, assembled from four named parts and nothing else.

VERDICT_LINES = "\n".join("  %-18s %s" % (v, G.VERDICT_HELP[v]) for v in G.VERDICTS)
SYSTEM = """You answer BILLING QUESTIONS from a timekeeper at a law firm, against ONE client's
def guidelines_block(ref):
def history_block(history):
def question_block(question):
def build(ref, question, history=(), doc_text=None):

src/answer.pyone turn

Assemble, call, parse. Nothing is repaired and nothing is coerced: a verdict spelled not-addressed is a wrong answer and is scored as one.

# One turn of the conversation: assemble, call, parse. This is the only file that spends money.

MAX_TOKENS = 32000
FENCE = re.compile(r"```(?:json)?\s*(.*?)```", re.S)
def parse(text):
def normalise(obj):
def ask(cfg, ref, question, history=(), complete=None, doc_text=None, max_tokens=None):
def converse(cfg, ref, turns, complete=None, doc_text=None, max_tokens=None):
def locates(ref, clause_ref, doc_text=None):

evals/baseline.pythe two free floors — a swap seam

BM25 over the clauses plus a verdict lexicon, and the null hypothesis that never re-reads on turn 2. Both free, both published, both scored by the same grader.

You change it to: A different retriever or a different verdict lexicon, scored by the same grader on the same 128 answers. The abstention threshold is a flag, and --sweep prints the whole curve rather than one tuned cell.

# THE FREE FLOOR. No key, no call, no cost -- and it is published whether it wins or loses.

DEFAULT_CUT = 0.20
LEXICON = [
def verdict_of(text):
class BM25:
def coverage(query, text):
def answer_one(bodies, headings, query, cut=DEFAULT_CUT):
def run_case(bodies, headings, case, mode="clause", cut=DEFAULT_CUT):

evals/judge.pythe grader

Pure code. Grounded = the governing clause cited AND its sentence quoted verbatim. Every rate carries its own denominator and none of them are meaned.

# Score a run. PURE CODE, NO MODEL, NO KEY, NO COST -- re-running this is free, forever.

NONE = {"", "none", "n/a", "na", "null", "-", "not applicable", "not addressed"}
WS = re.compile(r"\s+")
def _norm_ref(x):
def _flat(s):
def grade_answer(ans, gold, bodies, doc_refs):
def overlap_of(question, clause_text):
def _pct(n, d):
def score(cases, records, bodies_by_client, refs_by_client, overlap_cut):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 3874 input and 1235 output tokens per conversation (one question and one push-back, two calls), all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

MEASURED, FORCED AND PAIRED. This kit has two injection surfaces and one of them exists only because it is a conversation: the asker's own message goes into the same prompt as the guidelines, and the asker is the person whose time entry is about to be written off.

API_KEY is read from .env (gitignored) or the real environment; never logged, never placed in a prompt, and redacted from any provider error message the local UI passes to the browser.

The experimentWe attacked it — two gates

An indirect prompt injection has to clear two gates, and they behave differently. Reproduce with python3 -m evals.redteam --run-id x001-guideline-ask-flash --against r001-guideline-ask-flash --cases 16. Measured on 2026-08-25.

Both vectors are forced rather than seeded, and every reading is paired against its own un-injected answer from r001 rather than against gold -- a gold-relative count would score an already-wrong answer that stayed wrong as 'resisted'.

The resultThe vector that only exists because this is a CONVERSATION is twice as effective as the same words hidden in the document. Sent by the asker in the push-back turn, the injection was taken in 9 of 32 answers; planted immediately after the governing clause it was taken in 5 of 32. Overall 14 of 64 injected answers took some part of it and 12 of 64 (18.75 pct) cited clause 9.99 -- a clause number no document in this corpus prints -- quoting a sentence nobody wrote. 12 of the 14 landed on turn 2.

Both wordings ask for the same three things -- verdict billable, clause 9.99, and a sentence to quote -- so the whole answer is scored and not just the verdict. All 12 answers that cited 9.99 also quoted the injected sentence: taking the citation and taking the quote are the same event here, and neither happened without the other. The 16 probe cases are every one adverse in gold, so the injection is always pushing in the direction somebody would actually want it pushed. Two of the 14 counted as takes took only the VERDICT and landed on the document's own correct clause -- they were wrong before the injection and right after it -- which is why 12 of 64 is the strict figure and 14 of 64 the generous one.

HonestyWhat this does not prove

• Whether a real attacker would write either of these two sentences. Both were written by the kit's author against the kit's own answer contract.
• Whether an injection quoting a REAL clause number would be caught at all. Both wordings here demand a fake one, which the clause-membership check catches for free.
• Whether the deliberating tier resists differently. The probe used the fast tier only, and the two tiers already disagree measurably on ordinary judgement.
• Whether the rate holds at a different output ceiling. A sibling kit in this estate measured the identical readings suppressing at one ceiling and not another, so 18.75 pct is a property of a 32,000-token run and of nothing else.
• The app's HTTP surface. The probe drives src/answer.py directly, not src/app.py, so nothing here measures what a browser client could send.
• Whether the two vectors compound. They were fired separately, never together.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

Never write off time, approve a charge, release an invoice, bind the firm or give legal advice -- and never present any clause, client or guideline set in this corpus as real.

Stated to the model on every call in src/prompt.py's SYSTEM block, stated again in the notice on the local UI and at the head of every generated document, and true of the code by ABSENCE: there is no writer anywhere in src/, evals/, tools/ or ui/ outside results/*.json and data/.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• This is a PROMPT RULE PLUS AN ABSENCE, not a runtime enforcement layer. Nothing stops a forker adding a write path tomorrow.
• The injection result is a FLOOR on what an injector gets, not a ceiling: two wordings, one target clause, one placement each, one tier. 21.88 pct is not 'resistant'.
• Nothing here validates that the model's REASON is sound. The because field is unscored, because scoring prose would need a judge model and this kit has none.

WatchedWhat is watched, and why that one

The kit measures 58 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

12 measured by the latest run 46 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

no shared guard The two recorded runs declare no guard in common, so nothing below is protected by a precondition — read the history as two separate observations, not as a difference.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

8 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 4 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

comply · no model in the path — a baseline, not a peer column — 2 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 2 runs measured the same system — they differ on the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

comply · with the model in the path — 4 runs. Columns here are only ever compared with each other.

not a time series No two of these 4 runs measured the same system — they differ on clause_governs_answers, conversations, founded_pushbacks, rows, silent_topic_answers, unfounded_pushbacks, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 13 chips that all say so.

redteam · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 12 chips that all say so.

DeviationsWhat deviated

0 breaches across 8 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

on demand -- one call per turn, while the question is being asked

In one lineNo framework, deliberately

NO FRAMEWORK. A folder of readable Python, an empty requirements.txt and a prompt anyone can read end to end. A framework abstraction would own the retrieval step -- there is none here, deliberately -- and the conversation memory, which is a list of two tuples.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Linear: the document -> src/prompt.py -> src/adapters -> evals/judge.py, twice per conversation. No branching, no tool-calling and no agent loop.

The other sideWhat a framework costs you

• Swapping providers means editing the PROVIDERS dict by hand. In exchange requirements.txt stays empty, the prompt has no hidden templating layer, and the socket timeout is a constant a reader can see.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-guideline-ask-flash on the fast tier, 2026-08-25. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 5 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

2 runs not plotted. b001-guideline-ask-keyword, b002-guideline-ask-hold recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the rulebook is data read whole into the prompt.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT -- this repository's own licence, corpus included Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineGrounded -- the governing clause cited AND its sentence quoted verbatim

whether one answer names the clause that governs and quotes a sentence the document really prints, per answer, per turn

MethodHow the test was run

python3 -m evals.run --run-id <id> [--baseline clause|hold]; evals/judge.py compares strings. No model grades anything anywhere in this kit.

The inputOne real row, seen by every grader

The formulaWhat it computes

grounded = citation_exact AND quote_verbatim, over 128 answers. It is NEVER averaged with the push-back or abstention rates, which have denominators of 40, 24 and 32 and measure different things.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: this grader, against a gold clause and verdict DEALT by tools/build_corpus.py before the clause text was written -- never against another model's output. That deal is what makes the labels trustworthy and also what bounds them: the same generator wrote a residual clause into section 1 that arguably governs the 32 answers it labelled not_addressed.

How tight can the band be? 128 answers is the whole denominator, but the interesting sub-populations are small: 40 unfounded push-backs, 24 founded ones, 32 silent-topic answers. One flipped answer moves the founded-push-back rate by 4.2 points, so 95.83 pct against 91.67 between the two tiers is ONE row and not a result. The quote check has no threshold to sweep -- it is a whitespace-normalised substring test, and punctuation and capitals are deliberately not forgiven, because 'word for word' is the instruction the prompt gives.

Cadence: Re-run on any change to src/prompt.py, to data/topics.json, or to MAX_TOKENS in src/answer.py -- the first changes what is asked, the second changes what is asked ABOUT, and the third bounds whether a reply comes back at all. Also re-run after any change to src/guidelines.py: a defect in that clause parser is what evals/rescore.py exists for, and it silently mis-scores the quote column.

The decisionWhen to reach for it