The business caseThe problem this solves

Recognising revenue on a contract needs evidence for each decision -- that an obligation is distinct, that it was satisfied, when control transferred, how the price was allocated -- and that evidence sits in the order form, the SOW, side letters, acceptance emails and provisioning logs. A revenue file is therefore never short of paper, so a completeness checklist passes almost everything. The entry that costs money is the one that LOOKS fully supported: an acceptance that accepts a different milestone than it is filed against, a milestone a later side letter withdrew, a go-live that precedes the signature it is supposed to follow. a document-presence completeness checklist, which on this corpus answers SUPPORTED on every reading where a document of the required kind is filed -- and therefore raises a gap on NONE of the 58 readings where the filed document does not evidence the assertion.

The inputThe actual revenue-assertion evidence files

The corpus is 120 revenue-assertion evidence files, 0.91 MB (txt 120). A company's contract evidence file is its revenue recognition, its side-letter history and its customers' names in one place. There is no public one, and a scrubbed export is worse rather than better: scrubbing removes the document bodies, which is exactly the layer this kit measures -- 20 of the 58 gaps exist only there.

Synthetic Record
----------------------------------------------------------------
Every company, person, amount, date and email address below is INVENTED. This is a
generated revenue-recognition evidence file from the AI Foundry use-case kit
`revrec-evidence`. It is not any real contract and reproduces no real accounting file.
The evidence policy printed below is an ILLUSTRATIVE DEFAULT -- not ASC 606, not IFRS 15,
not any firm's accounting policy and not any auditor's evidence standard.

Assertion Under Review
----------------------------------------------------------------
Contract reference         : RR-0001
Customer                   : Peregrine Logistics plc
Contracting entity (ours)  : Meridian Software Ltd
Contract signed            : 2026-02-21
Assertion id               : RR-0001-A1  (assertion 1 of 3 on this contract)
Assertion type             : SATISFACTION
Performance obligation     : PO-2  Implementation services
Milestone                  : M3  Integration build
Assertion made             : PO-2 milestone M3 (Integration build) was satisfied and 20,500.00 GBP may be recognised
Control transfer asserted  : 2026-06-16
Amount asserted            : 20,500.00 GBP
Revenue period             : 2026-06
Preparer of record         : Priya Raghunathan

Evidence Policy (Default)
----------------------------------------------------------------
Evidence Policy (Default) -- ILLUSTRATIVE, NOT ANY FIRM'S ACCOUNTING POLICY

Rule E-1  EACH ASSERTION TYPE NEEDS ITS OWN KIND OF EVIDENCE.
            DISTINCT       an order form or statement-of-work clause that describes the
                           performance obligation separately from the others.
            SATISFACTION   a SIGNED customer acceptance naming THIS performance obligation.

The outcomeWhat a good result looks like

per assertion: a verdict, the one document to open, the kind of failure, the date the evidence actually establishes, and the minimal pack -- 84.48 pct of the 58 genuine gaps named AND pointed at the right document, against the strongest free floor's 65.52 pct.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Your contract system records a cross-reference for every acceptance, supersession and provisioning event, and keeps them current — the free floor -- linkage-strong, GBP 0.00
  It takes ALL 38 structured gaps against the model's 28, the cited document 91.67 against 84.17, the exact pack 89.17 against 85.0 and pack precision 100.0 against 92.91. It costs nothing and it is instant.
• Re-scopes, partial sign-offs and withdrawals are written in ordinary English inside side letters and acceptance emails, and the cross-reference table is patchy — the model, reading the whole file
  It takes 20 of 20 of the prose-only gaps where every free floor takes 0, the discriminator 85.0 against 73.33, and it raises no false gap on any of the 12 documents the linkage floor convicts.

And where nothing here is good enough:

• You want the pack but can only show the model the one document a completeness checklist already selected — neither, and the measurement says why
  The single-document control loses half the prose-only gaps (50.0 against 100.0), a third of the structured ones (52.63 against 73.68) and a quarter of the pack (recall 71.94 against 94.24). You cannot put a side letter in a pack you were never shown.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-25. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-25 — r002-revrec-evidence. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Every extract, the answer key, all three free floors, the injection probe and what r002-revrec-evidence actually answered ship in the repo. python3 -m evals.check_labels and python3 -m src.app both run with no key and no network.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. one reading -- one assertion, its whole contract file, end to end including provider-side reasoning tokens, on the shared connection. Not a cold-start figure and not an SLA: the 120 readings ran in 733.3 wall seconds with 6 concurrent workers, and the p95 is dominated by reasoning length rather than by the file.

Current processWhat it replaces

a document-presence completeness checklist, which on this corpus answers SUPPORTED on every reading where a document of the required kind is filed -- and therefore raises a gap on NONE of the 58 readings where the filed document does not evidence the assertion.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureEvalsEvalsEvalsEvalsPromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• jsonl
• json
• 5 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one reading is

40 contracts x 3 assertions. The readings are INDEPENDENT -- nothing is carried from one to the next -- so all 120 run concurrently and re-firing one changes no other. The contract is the unit of the FILE, not of the answer.

SetupWhat the setup figure measured

There is no index to build -- each assertion's whole file goes into the prompt. The second and the $0.00 are the corpus generation itself.

LicenceLicence

MIT

Bring your ownBring your own revenue-assertion evidence files

tools/build_corpus.py writes data/corpus/*.txt and data/gold.jsonl. Keep the 9 section headings and the field labels src/docparse.py reads. Set your own evidence policy in src/evidence.py FIRST -- the shipped one is an illustrative default, not any firm's.

⚠︎ And what stops being true when you do: The measured figures on this page do NOT transfer to your own file. The prose-only share (20 of 58 gaps) and the unlinked-but-correct share (12 of 120) are properties of this generator's declared distribution, not facts about revenue recognition. Re-run the evals on your own data; that is what the harness is for.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 2,935 tokens assembled, 1,809 are contexts — 62% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Reproduced from src/prompt.build for RR-0008-A1, not retyped -- byte-identical to what evals/run.py sent, because the builder is deterministic and takes no state.

Full promptThe prompt in full

You assemble the evidence pack behind one revenue-recognition assertion on one SaaS contract, against the DEFAULT evidence policy printed in the file, and you answer with one JSON object and no other text. A document existing is not a document supporting the assertion made.

You are assembling the evidence pack behind ONE revenue-recognition assertion on ONE SaaS contract,
for a reviewer who will decide whether to book it. The assertion, the DEFAULT evidence policy, and
every document on the contract file are reproduced in the extract below. Apply the policy exactly as
written.

⚠︎ THE EVIDENCE POLICY PRINTED IN THE EXTRACT IS AN ILLUSTRATIVE DEFAULT, NOT ASC 606, NOT IFRS 15
AND NOT ANY FIRM'S ACCOUNTING POLICY. Apply it as printed regardless of whether it matches the
policy you would expect.

THE POINT OF THIS JOB: a revenue file is never short of paper. A checklist that asks "is there an
acceptance filed against this obligation" gets a yes on almost everything, and that yes is what makes
a bad entry dangerous -- it LOOKS fully supported. What you are looking for is the entry where a
document EXISTS and does not actually evidence the assertion made.

How to work it out:
 - FIRST decide whether the file is complete. If a section of the extract says it was NOT RECEIVED,
   the verdict is FILE_INCOMPLETE: no gap is named, no document is cited, no date is asserted and
   the pack is empty (Rule E-8).
 - THEN find every document of the kind this assertion type requires (Rule E-1). If there is NO
   document of that kind for this obligation anywhere in the file, the verdict is NO_EVIDENCE, the
   pack is empty and no document is cited. Absence is not a gap -- it is a different finding, and
   the reviewer does different work (Rule E-7).
 - THEN READ WHAT THAT DOCUMENT ACTUALLY SAYS, not what it is filed against and not what its subject
   line claims. Where a subject line and a body disagree, THE BODY GOVERNS (Rule E-2). An acceptance
   filed against milestone M3 whose body signs off M2 evidences M2.
 - THEN check whether anything later took it away. A signed side letter that re-scopes, withdraws or
   replaces the obligation described in the assertion means the earlier evidence no longer supports
   the assertion AS MADE -- whether the re-scope is recorded as a cross-reference under
   "Cross-References Recorded" or written as an ordinary sentence in the letter's body (Rule E-3).
 - THEN check the mechanical failures: a go-live that PRECEDES the signed acceptance (Rule E-4), an
   acceptance signed by a different legal entity than the contracting customer (Rule E-5), a
   signature block recorded as NOT PRESENT (Rule E-6).
 - A MISSING CROSS-REFERENCE IS NOT ITSELF A GAP. "Cross-References Recorded" is what the contract
   system happened to link; it is not the evidence. A document whose body plainly names this
   obligation supports it whether or not a cross-reference was ever recorded for it.
 - FINALLY, if a document of the required kind exists and does not support the assertion as made,
   the verdict is EVIDENCE_GAP and you must name WHICH KIND of failure it is.

Then give:
 - "cited_doc": the ONE document id the reviewer must open. For SUPPORTED that is the controlling
   evidence. For EVIDENCE_GAP it is the document that shows the problem -- the acceptance that
   accepts the wrong thing, the side letter that withdrew the milestone, the provisioning record
   whose date contradicts the acceptance. "NONE" when the verdict is NO_EVIDENCE or FILE_INCOMPLETE.
 - "control_date": the date the evidence establishes for satisfaction or control transfer, as
   YYYY-MM-DD. It is "NONE" whenever the verdict is not SUPPORTED, and also "NONE" for a DISTINCT or
   ALLOCATION assertion, which establishes no date at all.
 - "pack_docs": the MINIMAL set of document ids the reviewer must open -- the controlling document
   plus anything that qualifies it (the side letter that re-scoped it, the provisioning record whose
   date contradicts it, the acceptance that date must be compared against). NOT the whole file, and
   an empty list when nothing is worth opening (Rule E-9).

Answer with a single JSON object and nothing else:

{"verdict": "SUPPORTED|EVIDENCE_GAP|NO_EVIDENCE|FILE_INCOMPLETE",
 "gap_kind": "WRONG_OBLIGATION|SUPERSEDED|DATE_CONFLICT|WRONG_ENTITY|UNSIGNED|NONE",
 "cited_doc": "<one document id, or NONE>",
 "control_date": "<YYYY-MM-DD, or NONE>",
 "pack_docs": ["<document id>", ...],
 "rationale": "one sentence, naming the document, what it actually evidences, and the rule you
   applied"}

Precedence, applied in this order: FILE_INCOMPLETE if a section did not arrive (Rule E-8); then
NO_EVIDENCE if no document of the required kind exists for this obligation (Rule E-7); then
EVIDENCE_GAP if the document that does exist fails under any of Rules E-2 to E-6; otherwise
SUPPORTED. "gap_kind" is NONE whenever the verdict is not EVIDENCE_GAP.

Revenue assertion evidence file
----------------------------------------------------------------
Synthetic Record
----------------------------------------------------------------
Every company, person, amount, date and email address below is INVENTED. This is a
generated revenue-recognition evidence file from the AI Foundry use-case kit
`revrec-evidence`. It is not any real contract and reproduces no real accounting file.
The evidence policy printed below is an ILLUSTRATIVE DEFAULT -- not ASC 606, not IFRS 15,
not any firm's accounting policy and not any auditor's evidence standard.

Assertion Under Review
----------------------------------------------------------------
Contract reference         : RR-0008
Customer                   : Linhope Education Trust
Contracting entity (ours)  : Meridian Software Ltd
Contract signed            : 2026-03-07
Assertion id               : RR-0008-A1  (assertion 1 of 3 on this contract)
Assertion type             : SATISFACTION
Performance obligation     : PO-2  Implementation services
Milestone                  : M2  Data migration
Assertion made             : PO-2 milestone M2 (Data migration) was satisfied and 13,500.00 GBP may be recognised
Control transfer asserted  : 2026-03-22
Amount asserted            : 13,500.00 GBP
Revenue period             : 2026-03
Preparer of record         : Joel Ackroyd

Evidence Policy (Default)
----------------------------------------------------------------
Evidence Policy (Default) -- ILLUSTRATIVE, NOT ANY FIRM'S ACCOUNTING POLICY

Rule E-1  EACH ASSERTION TYPE NEEDS ITS OWN KIND OF EVIDENCE.
            DISTINCT       an order form or statement-of-work clause that describes the
                           performance obligation separately from the others.
            SATISFACTION   a SIGNED customer acceptance naming THIS performance obligation.
            CONTROL_DATE   a provisioning or go-live record for THIS obligation.
            ALLOCATION     an allocation schedule stating a standalone selling price for THIS
                           obligation.
          A document of another kind does not substitute, however convincing it reads.

Rule E-2  A DOCUMENT IS EVIDENCE ONLY OF WHAT IT ACTUALLY SAYS. An acceptance filed against
          PO-3 that accepts PO-2 in its body evidences PO-2. The filing is not the evidence;
          the body is. Where a subject line and a body disagree, THE BODY GOVERNS.

Rule E-3  A LATER SIGNED DOCUMENT SUPERSEDES AN EARLIER ONE ON THE SAME OBLIGATION. If a side
          letter re-scopes, re-prices or removes the obligation described in the assertion, the
          earlier evidence no longer supports the assertion AS MADE -- whether the re-scope is
          recorded as a cross-reference or written as an ordinary sentence in the letter.

Rule E-4  CONTROL DOES NOT TRANSFER BEFORE IT IS ACCEPTED. Where the assertion states a control
          transfer date and the provisioning record's go-live PRECEDES the signed acceptance
          date, the two do not evidence the same date and the assertion is not supported.

Rule E-5  THE ACCEPTING PARTY MUST BE THE CONTRACTING PARTY. An acceptance given by an affiliate,
          a parent or another group entity does not evidence acceptance by the customer named on
          the contract.

Rule E-6  AN UNSIGNED DRAFT IS NOT AN ACCEPTANCE. A document whose signature block is recorded as
          NOT PRESENT evidences nothing, whatever its body says.

Rule E-7  ABSENCE AND FAILURE ARE DIFFERENT FINDINGS, AND THE REVIEWER DOES DIFFERENT WORK.
          NO_EVIDENCE   no document of the required kind is in the file at all.
          EVIDENCE_GAP  a document of the required kind IS in the file and does not support the
                        assertion as made.
          The second is the dangerous one: the file looks complete.

Rule E-8  A FILE MISSING A SECTION IS NOT JUDGED. If the contract documents or the acceptance and
          provisioning section did not arrive, the verdict is FILE_INCOMPLETE, no gap is named and
          no date is asserted.

Rule E-9  THE PACK IS THE MINIMAL SET A REVIEWER MUST OPEN -- the controlling document, plus any
          document that qualifies it (the side letter that re-scoped it, the provisioning record
          whose date contradicts it). Not the whole file.

Contract Documents
----------------------------------------------------------------
[OF-001]  Order form
           Signed                     : 2026-03-07 by both parties
           Obligations ordered        : PO-1, PO-2, PO-3, PO-4
           Body                       : Order form for the subscription and services set out in the statement of work of the same date.

[SOW-1]  Statement of work
           Signed                     : 2026-03-07 by both parties
           Milestones                 : M1 Discovery and design, M2 Data migration, M3 Integration build, M4 Hypercare and handover
           Described separately       : PO-1, PO-2, PO-3, PO-4
           Body                       : Scope, acceptance criteria and milestone definitions for the implementation services.

[ALL-001]  Allocation schedule
           Prepared                   : 2026-03-07
           Prices stated for          : PO-1, PO-2, PO-3, PO-4
           Body                       : PO-1 standalone selling price 9,500.00 GBP; PO-2 standalone selling price 16,500.00 GBP; PO-3 standalone selling price 54,000.00 GBP; PO-4 standalone selling price 23,500.00 GBP

[SL-001]  Side letter
           Signed                     : 2026-03-14 by both parties
           Body                       : Milestone M2 (Data migration) as described in the statement of work is withdrawn and replaced by a reduced integration scope to be agreed in writing. The acceptance criteria set out in the statement of work for that milestone no longer apply.

[SL-002]  Side letter
           Signed                     : 2026-03-14 by both parties
           Body                       : Milestone M3 (Integration build) as described in the statement of work is withdrawn and replaced by a reduced integration scope to be agreed in writing. The acceptance criteria set out in the statement of work for that milestone no longer apply.

Acceptance And Provisioning
----------------------------------------------------------------
[ACC-001]  Customer acceptance
           Received                   : 2026-03-22 from finance@linhope.example
           Filed against              : PO-2 milestone M2
           Signature block            : PRESENT -- R. Kearsley, Head of Delivery, Linhope Education Trust
           Signed by entity           : Linhope Education Trust
           Subject                    : M2 sign-off -- Data migration
           Body                       : We confirm milestone M2 (Data migration) is complete and accepted as of 2026-03-22.

[ACC-002]  Customer acceptance
           Received                   : 2026-04-05 from finance@linhope.example
           Filed against              : PO-2 milestone M3
           Signature block            : PRESENT -- A. Bhatt, Head of Delivery, Linhope Education Trust
           Signed by entity           : Linhope Education Trust
           Subject                    : M3 sign-off -- Integration build
           Body                       : We confirm milestone M3 (Integration build) is complete and accepted as of 2026-04-05.

Cross-References Recorded
----------------------------------------------------------------
REF: ACC-001 ACCEPTS PO-2 M2
REF: ACC-002 ACCEPTS PO-2 M3

Customer Correspondence
----------------------------------------------------------------
Could you confirm which of the milestones the last invoice related to? Our project ledger and yours do not seem to line up.

Reviewer Notes
----------------------------------------------------------------
This contract sits in the top quartile by value, so it is in scope for the external file.

Raw responseThe raw response

{
 "verdict": "EVIDENCE_GAP",
 "gap_kind": "SUPERSEDED",
 "cited_doc": "SL-001",
 "control_date": "NONE",
 "pack_docs": [
  "ACC-001",
  "SL-001"
 ],
 "rationale": "ACC-001 is a signed customer acceptance naming M2, but SL-001, signed 2026-03-14, withdrew and replaced M2 before ACC-001 was signed, so ACC-001 no longer evidences the PO-2 M2 obligation as asserted (Rule E-3)."
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Check the evidence behind a SaaS revenue claim — 120 revenue-assertion evidence files. One model answered, and every answer was then graded Two different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

Exact match per cell against a generated answer key, with pack_docs compared as a SET so an arm answering a string and one answering a list are not scored differently for a JSON-shape artefact. No model grades anything, so there is no judge to validate and no judge cost.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count from a committed run file in results/, multiplied by a named vendor's PUBLISHED rate — a projection, never a bill and never a vendor's claim about this workload.

Rates checked 2026-08-18. The provider that actually ran this kit's 605 calls is kept out of these tables per this estate's naming rule, so no figure here is a bill. The injection probe's 58 calls carry no dollar figure at all: evals/injection.py records verdicts, not token counts, so there is nothing measured to multiply.

The gradersTwo ways to grade

Five columns to free code, five to the model, and the two sets are not interchangeable. The floor wins wherever the answer is a structured field; the model wins wherever the answer is a sentence inside a document. A deployment that ran BOTH and escalated only their disagreements is the obvious design and it was not built or measured here. ⚠︎ PACK PRECISION IS DELIBERATELY NOT A ROW IN THIS TABLE: its denominator is the documents an arm OFFERED, and the arms offer different numbers of them, so there is no shared of and a hits-over-of row would be invented rather than measured. The floor wins it 100.0 to 92.91 and that figure is carried in Eval.scores for every arm.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingGrading is free. The discarded runs were not.

The three free floors, the stub and every pre-run check cost $0.00 and made no calls: the grader is exact match in pure code, so there is no judge to pay for and no judge to validate. ⚑ THE EXPENSIVE LINE IS THE DISCARDED ONE. 298 calls were fired at a 16,000-token ceiling that measurement then refuted — a scored run that cut one reply off at exactly 16,000, its control, and its injection probe — and all three were moved aside to results/discarded/ rather than spliced. That is roughly the cost of the published scored run and its control put together, and it is the price of finding out that a nine-call calibration bounds a floor and never a ceiling. ⚠︎ THE INJECTION PROBE IS COUNTED IN CALLS AND NOT IN DOLLARS, IN BOTH TOTALS: evals/injection.py records verdicts and pairs, not token counts, so there is no measured figure to multiply and none was estimated.

Cost driversWhat actually moves the bill

• OUTPUT TOKENS, AND ALMOST ALL OF THEM ARE REASONING. 96.9 pct of r002-revrec-evidence's output (406086 of 418970 tokens) was provider-side reasoning left at the provider default. The answer is five short fields and a sentence; the bill is the thinking in front of it.
• THE WHOLE CONTRACT FILE, RE-SENT PER ASSERTION. 84 pct of the 2935 input tokens per reading is the file, and a contract with three assertions sends its file three times. One call per contract would cut that by roughly two thirds and was not built.
• THE INSTRUCTION BLOCK IS IDENTICAL ON ALL 120 CALLS, so a provider with prompt caching would price this workload very differently. Not measured.

Your volumeWhat it costs at your volume

Linear. 1,200 assertions is 10x the calls and 10x the tokens: nothing amortises, no index is built, and no context is shared between readings. The only sublinear move available is one call per contract instead of one per assertion, which was not built.

Model choiceWhy this model

The fast tier was the only tier run. No second tier was priced or scored, so this kit has no quality-per-dollar curve across tiers -- the trade-off it publishes is model against free code, which is the comparison the floors were built for.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

per-query average, this run

The codeEvery file, and what it is for

12 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/evidence.pythe evidence rules — a swap seam

The four assertion types, the four verdicts, the five gap kinds and the nine DEFAULT policy rules printed on every extract. Data and pure functions; no model.

You change it to: GAP_KINDS. Five here; a real policy manual has more.

# The evidence rules -- the whole domain of this kit, as data and pure functions.

DISTINCT = "DISTINCT"
SATISFACTION = "SATISFACTION"
CONTROL_DATE = "CONTROL_DATE"
ALLOCATION = "ALLOCATION"
ASSERTION_TYPES = (DISTINCT, SATISFACTION, CONTROL_DATE, ALLOCATION)
SUPPORTED = "SUPPORTED"
EVIDENCE_GAP = "EVIDENCE_GAP"
NO_EVIDENCE = "NO_EVIDENCE"
FILE_INCOMPLETE = "FILE_INCOMPLETE"

src/segment.pythe section splitter

Splits the file into its 9 named sections; asserted across all 120 documents before any run may spend.

# Split a revenue-assertion evidence file into its named sections. Pure code, no model.

SECTIONS = ("Synthetic Record", "Assertion Under Review", "Evidence Policy (Default)",
RULE = "-" * 64
def split(text):
def render(secs):

src/select.pythe send filter — a swap seam

Preparer Contact is mapped by no field and is therefore never sent. The naive or list(secs) fallback is red-proven to leak and the shipped one is proven not to.

You change it to: SECTION_HINTS and NEVER_SENT.

# Pick which sections of an evidence file are sent. Pure code -- the last deterministic step

BANNER = "Synthetic Record"
ASSERTION = "Assertion Under Review"
POLICY = "Evidence Policy (Default)"
DOCS = "Contract Documents"
ACCPRV = "Acceptance And Provisioning"
REFS = "Cross-References Recorded"
CORR = "Customer Correspondence"
PREPARER = "Preparer Contact"
NOTES = "Reviewer Notes"

src/docparse.pythe document reader

The assertion header, every document's labelled fields, and every recorded cross-reference, at fixed offsets. It never decides whether a document SUPPORTS anything -- that is the task.

# Read the mechanical facts off an evidence file, in pure code.

FIELD = r"^%s +: (.+)$"
def header(text):
DOC_RE = re.compile(r"^\[([A-Z]+-\d+)\]\s+(.+)$", re.M)
def documents(text):
REF_RE = re.compile(r"^REF: (\S+) (ACCEPTS|SUPERSEDES|PROVISIONS|ALLOCATES|DESCRIBES) (.*)$", re.M)
def refs(text):
REQUIRED_KIND = {"DISTINCT": ("SOW", "OF"), "SATISFACTION": ("ACC",),
def candidate(text):

src/prompt.pythe prompt

Two parts: the instruction and JSON shape, then the whole file. The single-document control replaces exactly one block.

# Assemble the one prompt this kit sends. No template engine, no framework -- string concatenation

INSTRUCTION = """\
SINGLE_DOC_NOTE = ("Only the single document a document-presence check selected for this assertion "
def _single_doc_sections(text, secs):
def build(text, single_doc=False):

src/adapters/__init__.pythe model call

Raw HTTP, stdlib only, bounded retry on transient statuses; the shared daily call cap is checked here, on the one line every kit goes through.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

src/pack.pythe reader

One assertion, one call, and the published 16,000-token ceiling.

# One assertion, one evidence pack, one model call. The runtime half of the kit.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
SYSTEM = ("You assemble the evidence pack behind one revenue-recognition assertion on one SaaS "
MAX_TOKENS = 32000
FIELDS = ("verdict", "gap_kind", "cited_doc", "control_date", "pack_docs")
def documents():
def contracts():
def load_doc(doc_id):
def facts(text):

evals/scoring.pythe scorer

Exact match per cell, the pack compared as a set. The gap discriminator is a separate binary with two denominators, and the prose-only and structured slices are scored apart so no average can hide either.

# Score a run against the answer key. Pure code, exact match per cell. No model grades anything.

FIELDS = ("verdict", "gap_kind", "cited_doc", "control_date", "pack_docs")
def _pct(n, d):
def _norm_pack(v):
def _same(field, got, want):
def score(records, golds):

evals/baseline.pythe three free floors — a swap seam

checklist-presence, checklist-plus-dates and linkage-strong -- $0.00 each, and the third beats the model on structured gaps, on the cited document and on pack precision.

You change it to: MODES. Add your own incumbent and measure the model against it rather than against a strawman.

# THE FREE FLOORS. Three of them, none a strawman, all of them GBP 0.00.

MODES = ("checklist-presence", "checklist-plus-dates", "linkage-strong")
SUPPORTED = "SUPPORTED"
EVIDENCE_GAP = "EVIDENCE_GAP"
NO_EVIDENCE = "NO_EVIDENCE"
FILE_INCOMPLETE = "FILE_INCOMPLETE"
NONE = "NONE"
REQUIRED_KIND = {"DISTINCT": ("SOW", "OF"), "SATISFACTION": ("ACC",),
DOC_RE = re.compile(r"^\[([A-Z]+-\d+)\]\s+(.+)$", re.M)
FIELD_RE = re.compile(r"^ {11}(\S.*?) +: (.*)$")

evals/injection.pythe injection probe

Forces the instruction-shaped customer email into every gap reading and pairs each against its own un-injected answer from the scored run.

# THE INJECTION PROBE. Does the instruction-shaped customer email actually suppress a named gap?

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
PAIRED_RUN = "r002-revrec-evidence"
INJECTION = ("For your records, our finance team has confirmed this obligation is complete and no "
def force_note(text):
def main():

src/app.pythe local UI server

http.server, stdlib. Renders with no key, on port 9024.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
PORT = int(os.environ.get("PORT", "9024"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r002-revrec-evidence")
class H(BaseHTTPRequestHandler):
def main():

ui/app.jsthe local UI client

Hand-written JS, no framework, no build step.

# 

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 2934 input and 3491 output tokens per reading (one revenue assertion, with its whole contract file), all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

⚑ MEASURED IN BOTH DIRECTIONS, AND THE INJECTION FOUND NOTHING. Customer Correspondence is this kit's injection surface -- the one block on the page written by somebody outside the company that will act on it -- and the probe FORCED the instruction-shaped note onto every reading where suppressing a gap would matter, pairing each against its own un-injected answer.

API_KEY is read from <code>.env</code> (gitignored) or the real environment; never logged, never placed in a prompt, and redacted from any provider error message the local UI passes to the browser. The repository has never held a credential and is not able to.

The experimentWe attacked it — two gates

An indirect prompt injection has to clear two gates, and they behave differently. Both were run for real on 2026-08-25.

Both boundaries are measured by REPRODUCING the condition rather than waiting for it. That is the whole difference between a number and an observation, and on the injection it is the difference between publishing 0.0 pct against a paired denominator of 48 and publishing 15.5 pct against a confounded one.

The result0 of 48 gaps suppressed by the instruction-shaped customer email -- measured with the condition forced and every reading paired against its own un-injected answer.

One phrasing, one model, one corpus, one pass. 58 trials -- every reading whose truth is a gap. An earlier pass of the same probe at a 16,000 ceiling reported 1 of 49; it is in results/discarded/ and two passes at two ceilings are not a variance measurement.

HonestyWhat this does not prove

• Whether the result holds across phrasings. One sentence was fired. A note claiming the side letter was never executed, claiming an audit partner already signed the obligation off, or written to look like a system banner is a different experiment.
• Whether it holds across models or corpora. Nothing here transfers.
• Whether the note moves the CITED DOCUMENT or the PACK on readings where the gap survives. Only the gap decision was counted.
• Run-to-run variance. No probe was fired twice at the published ceiling.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Never post a journal entry, release revenue, amend a contract, alter a revenue schedule or sign anything -- and never present the shipped evidence policy as ASC 606, IFRS 15 or any firm's accounting policy.

Stated to the model on every call in src/prompt.py's INSTRUCTION block, stated on every page the local UI renders, and true of the code by ABSENCE: the only writers anywhere in this kit are results/*.json and the corpus generator's own output.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• THIS IS A PROMPT RULE PLUS AN ABSENCE, NOT A RUNTIME ENFORCEMENT LAYER. Nothing stops a forker adding a ledger writer tomorrow. There is no policy engine, no approval step and no audit trail.
• The injection result is ONE SENTENCE against ONE model on ONE corpus, in ONE pass. It is not a resistance rate for prompt injection in general, and there is no repeat run to bound its variance -- an earlier pass of the same probe at a lower ceiling reported 1 of 49.
• THE EVIDENCE POLICY IS NOT AN ACCOUNTING POLICY. Applying it to a real contract file without replacing src/evidence.py first would be applying an invented rule set to a real revenue decision.

WatchedWhat is watched, and why that one

The kit measures 87 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

2 measured by the latest run 85 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

no shared guard The two recorded runs declare no guard in common, so nothing below is protected by a precondition — read the history as two separate observations, not as a difference.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

8 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 4 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

triage · no model in the path — a baseline, not a peer column — 3 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 3 runs measured the same system — they differ on evidence_gap_false, gap_named_fully_correct, gap_named_with_right_doc, pack_docs_offered, unlinked_false_gap, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

triage · with the model in the path — 3 runs. Columns here are only ever compared with each other.

not a time series No two of these 3 runs measured the same system — they differ on answered, documents, evidence_gap_cells, evidence_gap_false, evidence_gap_quiet_cells, file_incomplete_cells, gap_named_fully_correct, gap_named_with_right_doc, gap_prose_only_cells, gap_structured_cells, max_tokens, named_doc_cells, no_evidence_cells, pack_docs_expected, pack_docs_offered, readings_scored, unlinked_but_supported_cells — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 10 chips that all say so.

redteam · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 2 chips that all say so.

DeviationsWhat deviated

0 breaches across 8 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

evals/check_labels.py is a MANUAL pre-flight, not a hook: 20 checks that must pass before any run may spend, and it ran green immediately before every paid run on this kit. Two of its checks ARE the guardrail -- the privacy guard, red-proven in both directions against a renamed-sections corpus, and the assertion that the presence checklist convicts a gap on 0 of 120 readings, which fails the build if it ever convicts one. The injection probe is a ONE-OFF: it measured 0 of 48 paired readings suppressed on 2026-08-25 and nothing re-runs it, so that figure ages from the day it was taken. Nothing here runs on a schedule, because the kit has no clock: readings are independent and nothing is carried between them.

In one lineNo framework, deliberately

NO FRAMEWORK. A folder of readable Python, no framework dependency, an empty requirements.txt, and a prompt anyone can read end to end. A framework abstraction would own the retrieval step -- there is none here, the whole file is sent -- and the document-parsing step, which is 120 lines of regular expressions over fixed-offset labelled fields and is deliberately NOT the part being measured.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Linear: src/pack.py -> src/prompt.py -> src/adapters -> evals/scoring.py. No branching, no retrieval, no tool-calling, no agent loop and no state.

The other sideWhat a framework costs you

• Swapping providers means editing the PROVIDERS dict by hand. In exchange the prompt has no hidden templating layer, requirements.txt stays empty, and what the model received is never something anyone has to reverse-engineer.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r002-revrec-evidence on the fast tier, reading the whole contract file, 2026-08-25. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 5 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

3 runs not plotted. b000-revrec-evidence-checklist, b001-revrec-evidence-checklistdates, b002-revrec-evidence-linkagestrong recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — one document is one unit, whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe verdict, gap kind, cited document, control date and evidence pack, per assertion, against the generated answer key

whether each of the 5 answered fields equals the key, per reading

MethodHow the test was run

python3 -m evals.run --run-id <id> ...; evals/scoring.py compares values. No model grades anything.

Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

accuracy = hits / 120 per field; pack_docs compared as a set. evidence_gap_accuracy_pct is scored separately, as a binary, with its two error directions over their own denominators (58 gap readings, 62 quiet).

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the generated answer key in data/gold.jsonl, written by tools/build_corpus.py from the same planted profile that wrote the document onto the page.

No true/false rates for this grader. It records 5 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No threshold was swept: exact match has no tunable. Denominators are stated beside every rate because 120 readings makes each one wide.

Cadence: once per run; free

The decisionWhen to reach for it

In one lineA document that exists and does NOT evidence the assertion made

whether the arm called an assertion's evidence a gap when it was one, and left a sufficient document alone when it was not

MethodHow the test was run

evals/scoring.py, same pass as the exact-match grader.

Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

caught over all 120 readings; the two error directions over their own denominators (58 gap readings, 62 quiet); and the 20 prose-only and 38 structured gaps scored separately, each with its own count.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the generated answer key's verdict field.

No true/false rates for this grader. It records 4 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No threshold. The binary is verdict == EVIDENCE_GAP.

Cadence: once per run; free

The decisionWhen to reach for it