The business caseThe problem this solves

A marketing piece does not arrive as a list of claims. It arrives as a page: a headline, a table, a paragraph about the strategy, and a block of small print at the bottom. Somebody has to turn it into claims, decide which review item governs each one, and then check the thing that is genuinely tedious -- whether the note that item asks for is beside the claim, or merely somewhere on the page. Those are not the same question, and the second one is what gets ticked when a reviewer is reading their eleventh proof of the afternoon. Someone opening each proof of a client-facing piece, listing every claim it makes, deciding which checklist item governs each one, and then -- the slow part -- checking whether the note that item asks for is actually beside the claim rather than merely somewhere on the page.

The inputThe actual marketing pieces

The corpus is 50 marketing pieces, 0.27 MB (txt 50). Plain text, one format, invented rather than fetched -- a real marketing piece names a real firm, a real fund, real money and a real track record, carries the firm's branding, and the version that went to review is commercially sensitive in a way the published version is not. There is no public corpus of (piece, correct review worksheet) pairs. Generating it also makes the label mechanical: gold's two calls are the shipped checklist's own lookup over the same four facts the piece states, never somebody's reading of whether a piece feels compliant.

WRAYBURN CAPITAL
MARKETING REVIEW EXTRACT — GENERATED FOR THIS KIT, NOT A REAL DOCUMENT

Piece
-----

  Piece reference:    MP-2023-587
  Extract id:         MRV-0001
  Piece kind:         quarterly client email
  Piece date:         2023-09
  Intended audience:  institutional prospects

  This extract is prepared for marketing review. The Schedule Of Claims below lists the coded
  claims THIS VERSION of the piece makes. The Claim blocks that follow set each of them out.

Distribution Record
-------------------

  Approver:               R. Ostrander
  Internal approval ref:  AP-2023-6492
  Mailing list:           adviser-panel-tier1
  Print run:              18,500
  Intended distribution:  2023-11
  Sales contact:          N. Braybrooke, direct line 01390 458068

Schedule Of Claims
------------------

  ------------------------------------------------------------------------------------------
  CODE        KIND                      SUMMARY
  ------------------------------------------------------------------------------------------
  MC-7234     performance_figure        Bramblewood Global Equity Fund's stated return
  MC-8033     superlative               Harewell Growth Fund ranked against its sector
  MC-3499     forward_statement         What Stonecross Income Fund is said to deliver next
  MC-5321     service_description       What Fennimore Sustainable Fund does
  ------------------------------------------------------------------------------------------

Performance Summary
-------------------

  ------------------------------------------------------------------------------------------
  CLAIM       BASIS        FUND OR STRATEGY                    RETURN     BENCH
  ------------------------------------------------------------------------------------------

The outcomeWhat a good result looks like

A worksheet row per claim -- the claim code, its summary, the kind the piece states, who the piece says is making it, where the note its item asks for actually sits, and what period it is about -- with two calls attached: does the claim carry its disclosure where the checklist asks for it, and is the claim historical, forward-looking or neither. Plus one flag taken afterwards in pure code. It produces a REVIEWER'S WORKSHEET and never an approval or a rejection. It does not approve a piece, reject one, sign one off, clear one for distribution, file anything or record a sign-off. NO PIECE IS APPROVABLE BY THIS KIT: a worksheet whose every row reads carried is a piece against which this kit's own invented checklist found nothing, which is a much smaller statement than an approval.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Preparing a batch of client-facing proofs for a reviewer before they open them — the one tier measured here -- 263 of 275 disclosure calls, 275 of 275 horizon calls, and 0 of 102 claims waved through
  0 pct waved through against the free scan-line floor's 67.65 pct is the whole gap, and it is measured on the direction that costs something: a claim ticked without its disclosure is a claim nobody looks at again. It also left every one of the 39 decoy codes off, against the floor's 13 phantom claims.

And where nothing here is good enough:

• Deciding whether a cheaper or a more deliberate model would do better here — nothing -- this page cannot answer it
  Only one model was run. There is no tier comparison, no repeatability pair and no ablation on this page, and the horizon grader is already at 275 of 275 so it would not separate two models anyway.
• Deciding whether a piece may be distributed — nothing here, ever
  NO PIECE IS APPROVABLE BY THIS KIT. Every grader on this page measures agreement with data/rulebook.json, which is an invented checklist. A worksheet whose every row reads carried means this kit's own construction found nothing, which is not a statement about any rule that binds anybody.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-22. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-22 — r001-marketing-review. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Checked on a fresh checkout with API_KEY left blank: python -m src.app renders the page, the corpus and the shipped run records are browsable offline, and Build worksheet returns a plain sentence saying nothing was called. python -m evals.check_labels and both free runs (--stub, --baseline) need no key and no network.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per marketing piece

Current processWhat it replaces

Someone opening each proof of a client-facing piece, listing every claim it makes, deciding which checklist item governs each one, and then -- the slow part -- checking whether the note that item asks for is actually beside the claim rather than merely somewhere on the page.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 0 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one section is

cut on underlined section headings; a piece with none falls back to one whole-document segment. On this kit the cut is part of the ANSWER, not just a way of finding one -- note_placement asks which block a note sits in

SetupWhat the setup figure measured

There is no index. Preparation is segmentation only -- 50 pieces cut into 621 sections, of which 521 are sent and 100 are not: the 50 Distribution Records and the 50 header banners above them, neither of which any field maps to.

LicenceLicence

MIT -- this repository's own licence. Every firm, fund, strategy, benchmark, ranking body, data provider, person, figure and piece reference is invented; nothing was fetched and no real marketing piece was read

Bring your ownBring your own marketing pieces

REPLACE data/rulebook.json FIRST, and on this kit that is not advice, it is a precondition. The shipped checklist is invented here, it is not any regulator's rule or any firm's manual, and running real marketing pieces against it would be running them against a rule nobody qualified wrote. Then replace data/corpus/*.txt, write your own data/fields.json, and supply a gold row per piece. SECTION_HINTS in src/select.py maps fields to section headings and WILL need editing; when it does not match, selection falls back to the whole document -- slower, more expensive, always correct, and able to smuggle a section you meant to withhold, which is what evals/check_labels.py caught on this kit's first run.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 4,920 tokens assembled, 2,834 are contexts — 58% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured token-for-token via evals/prompt_tokens.py's nested-prefix method on MRV-0034 -- four calls at max_tokens=1, each part's size being the difference between two consecutive prompt_tokens counts the provider itself returned, never a character estimate. Committed at results/tokens-p001-marketing-review.json.

Full promptThe prompt in full

You read a CLIENT-FACING MARKETING PIECE -- a factsheet, a pitch page, a client email -- and return a REVIEWER'S WORKSHEET of the claims in it. You return JSON and nothing else.

You are preparing a worksheet for a qualified reviewer. You never approve a piece, never reject one, never sign one off, never clear one for distribution, never file anything and never record a sign-off. NOTHING YOU RETURN MAKES A PIECE APPROVABLE. Your job is to say what the piece states, apply the checklist given below, and NAME THE CLAIMS THE CHECKLIST DOES NOT SETTLE.

RULES, in order of importance:
1. ONE OBJECT PER CLAIM IN THIS VERSION OF THIS PIECE, AND NO OTHERS. A claim withdrawn by the Review Notes is not made by this version, even though its whole block is still printed. A specimen in the House Style Specimen appendix is an example shown to staff and is made to nobody. A claim carried in the previous version and referred to here for context belongs to that version. All three carry codes in the same format as a real claim, and putting one on the worksheet puts a claim in front of a reviewer that this piece does not make.
2. REPORT WHAT THE PIECE STATES. `claim_kind`, `asserted_by`, `note_placement` and `time_frame` are readings of the piece, not judgements about it. `claim_kind` is copied verbatim from the Kind line of the claim's own block.
3. `note_placement` IS THE HARDEST FIELD AND THE ONE TO SLOW DOWN ON. Every note in a piece is printed as `Note for MC-nnnn: ...`, naming the claim it accompanies. For each claim, find the checklist item that governs its kind, read what that item REQUIRES, and then look for a note tagged with THIS claim's code that actually says that. If it is inside this claim's own block, 'in_this_block'. If it is somewhere else in the piece -- the Important Information footer, the note under the Performance Summary table, or inside a DIFFERENT claim's block -- 'elsewhere_in_piece'. If no note tagged to this claim says what the item requires, 'not_in_this_piece'. A NOTE TAGGED TO THIS CLAIM THAT BELONGS TO A DIFFERENT ITEM DOES NOT COUNT: a note about periods that have already ended does not cover a modelled figure, however adjacent it is.
4. `disclosure` IS DECIDED ONLY BY THE CHECKLIST BELOW, from `claim_kind`, `asserted_by` and `note_placement`. Work through the seven steps IN ORDER and STOP at the first that fires:
   a. asserted_by 'third_party' -> 'not_required'.
   b. a kind the checklist has no item for -> 'not_determined'.
   c. asserted_by 'not_stated' -> 'not_determined'.
   d. note_placement 'in_this_block' -> 'carried'.
   e. note_placement 'elsewhere_in_piece' AND the item lets the note sit anywhere -> 'carried'.
   f. note_placement 'elsewhere_in_piece' AND the item wants the note beside the claim -> 'not_carried'.
   g. note_placement 'not_in_this_piece' -> 'not_carried'.
5. THE NOTE BEING SOMEWHERE ON THE PAGE IS NOT THE SAME AS THE ITEM BEING MET. Step (f) is the one a confident reader skips: the words are there, in the footer or under the table or beside another claim, and the item asks for them beside THIS claim. On this checklist that is the wrong answer rather than a defensible one.
6. `not_required` AND `not_determined` ARE DIFFERENT ANSWERS AND YOU ARE EXPECTED TO USE BOTH. `not_required` means the checklist was worked through and lands on nothing being needed. `not_determined` means the checklist does not settle it at all -- and a worksheet that never reaches for it is guessing, because a claim recorded as settled is a claim nobody re-reads.
7. `horizon` IS DECIDED ONLY FROM `time_frame`: 'period_ended' -> 'historical', 'future_outcome' -> 'forward_looking', 'no_period' -> 'not_determined'. It is asked of every claim independently of its disclosure call. READ THE PERIOD, NOT THE TENSE.
8. Copy `claim_code`, `claim_label` and `claim_kind` verbatim from the piece. Use the exact allowed value for every field that lists them, and return every field for every claim.

REVIEW CHECKLIST (the authority for `disclosure` and `horizon`; this is an ILLUSTRATIVE checklist written for this kit -- it is not any regulator's rule, any supervisor's guidance, any trade body's code or any firm's compliance manual, and the item codes were invented here)

WHAT COUNTS AS A CLAIM ON THE WORKSHEET
  - A claim is on the worksheet if and only if it is a coded claim in THIS VERSION of THIS PIECE - a row of the piece's Claim Schedule.
  - A claim withdrawn from this version by the review notes is NOT on the worksheet, even though its code and its whole block are still printed in the piece.
  - A specimen claim printed in the House Style Specimen appendix is NOT on the worksheet. It is an example of how to write a claim, shown to staff. It is not addressed to any client and it is not made by the piece.
  - A claim carried in an EARLIER approved version of this piece and referred to here for context is NOT on the worksheet. It is a different piece's paperwork.

THE CHECKLIST'S ITEMS -- one governs each claim, chosen by the claim's KIND
  MRC-1   performance_figure     note must be adjacent  -- a note saying the figures are for periods that have ended and do not indicate what will be returned in future
  MRC-2   hypothetical_figure    note must be adjacent  -- a note saying the figures were modelled rather than earned, that no client account was managed on that basis over the period shown and that no investor received them
  MRC-3   forward_statement      note must be adjacent  -- a note saying the outcome is not assured and may not be achieved
  MRC-4   superlative            note must be anywhere  -- a note naming the source of the ranking, the date it was awarded and the population it covered
  MRC-5   fee_basis              note must be anywhere  -- a note saying which charges the figure is stated after and which it is stated before
  MRC-6   track_record           note must be adjacent  -- a note saying what period the record covers and that it includes every account managed on the strategy over it
  (no item)  service_description    the checklist covers no claim of this kind

THE FOUR STATED FACTS, read off the piece
  claim_kind:
    performance_figure     the piece's Claim Schedule labels this claim a stated return figure for a period that has ended
    hypothetical_figure    the schedule labels it a modelled, simulated or backtested figure
    forward_statement      the schedule labels it a statement about a future outcome
    superlative            the schedule labels it a ranking, award or best/largest/leading claim
    fee_basis              the schedule labels it a figure stated after or before a charge
    track_record           the schedule labels it a claim about the length or continuity of a record
    service_description    the schedule labels it a plain description of what the firm or fund does
  asserted_by:
    firm                   the claim's own block states the firm as the source, or states no source other than the firm itself
    third_party            the block names an external source and reports the claim AS that source's - a ranking awarded by a named body, a figure published by a named data provider
    not_stated             the block attributes the claim to nobody at all, naming neither the firm nor any external source
  note_placement:
    in_this_block          the note this claim's own item requires appears inside this claim's block
    elsewhere_in_piece     that note appears somewhere else in the piece - the Important Information footer, the note under the performance table, or inside a DIFFERENT claim's block
    not_in_this_piece      that note appears nowhere in the piece. A note belonging to a DIFFERENT item sitting beside this claim does not change this value: it is not the note this item requires
  time_frame:
    period_ended           the claim is about a period that has already ended
    future_outcome         the claim states an outcome in a period that has not happened yet, HOWEVER it is worded - a sentence in the past or present tense that says what will be delivered is a future outcome
    no_period              the claim names no period at all

THE DISCLOSURE CALL -- work through IN ORDER, stop at the first that fires
  1. REPORTED, NOT ASSERTED. If asserted_by is `third_party`, the worksheet records `not_required`. The items in this checklist attach to claims the FIRM makes; a ranking or a figure the piece reports as a named external source's, and attributes to that source, is not one. This is settled, and it is not the same answer as `not_determined`.
  2. NO ITEM COVERS IT. If claim_kind is one this checklist has no item for, the worksheet records `not_determined`. Nothing in the checklist says what must accompany that kind of statement, so the checklist does not settle it. THIS IS A FINDING ABOUT THE CHECKLIST, not about the piece, and it is the row a reviewer most needs to see - it is where their own judgement is required and no list is going to help them.
  3. NOBODY IS NAMED. If asserted_by is `not_stated`, the worksheet records `not_determined`. Step 1 turns on whose claim it is and the piece does not say, so the checklist cannot be worked through to an answer.
  4. THE NOTE IS WITH THE CLAIM. If note_placement is `in_this_block`, the worksheet records `carried`.
  5. THE ITEM ALLOWS IT ANYWHERE. If note_placement is `elsewhere_in_piece` AND the governing item's placement is `anywhere`, the worksheet records `carried`. Two items in this checklist are written that way - MRC-4 and MRC-5 - because a source line and a charges line are read once for the whole piece.
  6. THE NOTE IS IN THE PIECE AND STILL NOT WITH THE CLAIM. If note_placement is `elsewhere_in_piece` AND the governing item's placement is `adjacent`, the worksheet records `not_carried`. THIS IS THE STEP A CONFIDENT READER SKIPS. The note is on the page. It is in the footer, or under the table, or beside a different claim. The item asks for it beside THIS claim and it is not there, so on this checklist the item is not met - and a reader scanning for whether the words appear at all will tick it.
  7. THE NOTE IS NOT IN THE PIECE. If note_placement is `not_in_this_piece`, the worksheet records `not_carried`. A note belonging to a different item sitting right beside the claim does not save it: a modelled figure is not covered by a note about periods that have ended, however adjacent that note is.

THE HORIZON CALL
  1. If time_frame is `period_ended`, the worksheet records `historical`.
  2. If time_frame is `future_outcome`, the worksheet records `forward_looking`. WORDING DOES NOT DECIDE THIS. A sentence written in the past or the present tense that states what a strategy will deliver over a period that has not happened is a forward-looking statement, and reading the tense instead of the period is the commonest way to get this column wrong.
  3. If time_frame is `no_period`, the worksheet records `not_determined`. The horizon question is asked of every claim independently of the disclosure call - a claim whose note is perfectly placed still either names a period that has ended, a period that has not, or none at all.

WHY `not_determined` IS A REAL ANSWER
  A worksheet that never says `not_determined` is not a thorough worksheet, it is a guessing one. The whole value of this shape is that a reviewer can see, in one column, which claims the checklist actually settles and which ones need a person to think. Over-confidence here costs more than caution in both directions: a claim recorded as settled is a claim nobody re-reads, and a checklist stretched over a claim it does not cover produces a tick that means nothing. `not_required` is a different answer again - it means the checklist WAS worked through and it lands on nothing being needed.

Return these:
- piece_ref (string) -- the piece reference from the Piece section, verbatim
- claims (array of objects) -- one object per claim THIS VERSION makes, each carrying:
    - claim_code (string) -- the claim code from the Claim Schedule row, verbatim (for example MC-4417). THIS IS THE KEY -- one object per claim in THIS VERSION of this piece and no others
    - claim_label (string) -- the claim's short description, copied verbatim from the Summary line of the claim's own Claim block
    - claim_kind (enum) one of: performance_figure, hypothetical_figure, forward_statement, superlative, fee_basis, track_record, service_description -- the claim kind, copied verbatim from the Kind line of the claim's own Claim block. The piece states it; you are not asked to re-classify the claim
    - asserted_by (enum) one of: firm, third_party, not_stated -- who the claim's own block states is making the claim. 'third_party' when the block names an EXTERNAL source -- a ranking body, a data provider, a research house -- and reports the claim as that source's. 'firm' when the block states the firm as the source, or states no source other than the firm itself. 'not_stated' when the block names nobody at all, neither the firm nor any external source. Read the block; never infer an attribution from how confident the claim sounds
    - note_placement (enum) one of: in_this_block, elsewhere_in_piece, not_in_this_piece -- WHERE THE NOTE THIS CLAIM'S OWN CHECKLIST ITEM REQUIRES APPEARS -- and this is the hardest field on the worksheet. 'in_this_block' when that note is inside this claim's own Claim block. 'elsewhere_in_piece' when it appears somewhere else in the piece: the Important Information footer, the note printed under the Performance Summary table, or inside a DIFFERENT claim's block. 'not_in_this_piece' when that note appears nowhere in the piece. ⚠ A NOTE BELONGING TO A DIFFERENT ITEM SITTING RIGHT BESIDE THIS CLAIM DOES NOT COUNT -- read what the note actually says against what this claim's item requires. A note about periods that have ended does not cover a modelled figure, however adjacent it is. Where the checklist has no item for the claim's kind, there is no such note and the value is 'not_in_this_piece'
    - time_frame (enum) one of: period_ended, future_outcome, no_period -- the PERIOD the claim is about. 'period_ended' when it is about a period that has already finished. 'future_outcome' when it states an outcome in a period that has not happened yet -- HOWEVER IT IS WORDED. A sentence in the past or present tense that says what a strategy will deliver over the coming years is a future outcome; reading the tense instead of the period is the commonest way to get this wrong. 'no_period' when the claim names no period at all
    - disclosure (enum) one of: carried, not_carried, not_required, not_determined -- the worksheet's disclosure call for this claim, decided STRICTLY by the shipped checklist from `claim_kind`, `asserted_by` and `note_placement` and nothing else. Work through the seven steps IN ORDER, stopping at the first that fires. (1) asserted_by 'third_party' -> 'not_required'. (2) a kind the checklist has no item for -> 'not_determined'. (3) asserted_by 'not_stated' -> 'not_determined'. (4) note_placement 'in_this_block' -> 'carried'. (5) note_placement 'elsewhere_in_piece' AND the item allows the note anywhere (MRC-4, MRC-5) -> 'carried'. (6) note_placement 'elsewhere_in_piece' AND the item requires the note beside the claim -> 'not_carried'. (7) note_placement 'not_in_this_piece' -> 'not_carried'. THE NOTE BEING SOMEWHERE ON THE PAGE IS NOT THE SAME AS THE ITEM BEING MET: step 6 is the one a confident reader skips
    - horizon (enum) one of: historical, forward_looking, not_determined -- the worksheet's horizon call, decided STRICTLY from `time_frame`: 'period_ended' -> 'historical', 'future_outcome' -> 'forward_looking', 'no_period' -> 'not_determined'. Asked of every claim independently of its disclosure call

Return a JSON object with exactly these top-level keys: piece_ref, claims
`claims` is an array. Return it empty only if this version of the piece makes no claim at all.

MARKETING PIECE
---------------
Piece
-----

  Piece reference:    MP-2024-134
  Extract id:         MRV-0034
  Piece kind:         campaign landing page
  Piece date:         2024-02
  Intended audience:  existing clients

  This extract is prepared for marketing review. The Schedule Of Claims below lists the coded
  claims THIS VERSION of the piece makes. The Claim blocks that follow set each of them out.

Schedule Of Claims
------------------

  ------------------------------------------------------------------------------------------
  CODE        KIND                      SUMMARY
  ------------------------------------------------------------------------------------------
  MC-8162     performance_figure        Fennimore Global Equity Fund's stated return
  MC-4032     hypothetical_figure       Windermarsh Absolute Return strategy's modelled return
  MC-2150     superlative               Kirtlow Diversified Fund ranked against its sector
  MC-8629     performance_figure        Rossmere Balanced Fund's stated return
  MC-7606     forward_statement         What Fennimore Balanced Fund is said to deliver next
  ------------------------------------------------------------------------------------------

Performance Summary
-------------------

  ------------------------------------------------------------------------------------------
  CLAIM       BASIS        FUND OR STRATEGY                    RETURN     BENCH
  ------------------------------------------------------------------------------------------
  MC-8162     actual       Fennimore Global Equity Fund          6.4%      1.5%
  MC-4032     modelled     Windermarsh Absolute Return stra      5.7%      4.0%
  MC-8629     actual       Rossmere Balanced Fund                3.4%      5.6%
  ------------------------------------------------------------------------------------------

Claim MC-8162
-------------

  Summary: Fennimore Global Equity Fund's stated return
  Kind: performance_figure

  Fennimore Global Equity Fund investors saw a return of 6.4 per cent a year across the 2021 to
  2023 period, compared with 1.5 per cent for the Northbank Composite. That is the return the
  Fennimore Global Equity Fund will go on delivering through to 2028.
  The years in question are 2024 through 2028.
  Note for MC-8162: The figures shown for this claim are for periods that have already ended.
  They are not an indication of what will be returned in any future period.

Claim MC-4032
-------------

  Summary: Windermarsh Absolute Return strategy's modelled return
  Kind: hypothetical_figure

  A model portfolio run on the Windermarsh Absolute Return strategy across 2019 to 2023 produces
  5.7 per cent a year, against 4.0 per cent for the Northbank Composite.
  This claim is made by Holmsdale Investment Management on the basis of its own records.
  The period covered closed before this piece was written.

Claim MC-2150
-------------

  Summary: Kirtlow Diversified Fund ranked against its sector
  Kind: superlative

  The Kirtlow Diversified Fund is the top-rated fund of its kind.
  The figure is published by Pentworth Data and is reproduced here as that provider's, not as a
  claim by Holmsdale Investment Management.

Claim MC-8629
-------------

  Summary: Rossmere Balanced Fund's stated return
  Kind: performance_figure

  Over the 2021 to 2023 period the Rossmere Balanced Fund grew by 3.4 per cent a year, ahead of
  the Northbank Composite at 5.6 per cent.
  The figure is published by Verrell Research and is reproduced here as that provider's, not as
  a claim by Holmsdale Investment Management.
  The record above runs to the end of 2023.
  Note for MC-8629: The figures shown for this claim are for periods that have already ended.
  They are not an indication of what will be returned in any future period.

Claim MC-7606
-------------

  Summary: What Fennimore Balanced Fund is said to deliver next
  Kind: forward_statement

  The Fennimore Balanced Fund delivers 10.8 per cent a year across a full cycle, and the cycle
  running from 2024 to 2027 is no exception.
  The years in question are 2024 through 2027.

Important Information
---------------------

  This piece is issued by Holmsdale Investment Management. It is addressed to existing clients.
  The value of an investment and the income from it can fall as well as rise.
  Note for MC-2150: The ranking in this claim was awarded by the source named, on the date
  shown, on that source's own criteria, and covered only the firms that entered.

END OF EXTRACT MRV-0034

Prepared for marketing review. This extract is generated and describes no real firm, fund,
strategy, benchmark, ranking or person. Nothing in it approves, clears or rejects anything.

Raw responseThe raw response

{"piece_ref":"MP-2024-134","claims":[{"claim_code":"MC-8162","claim_label":"Fennimore Global Equity Fund's stated return","claim_kind":"performance_figure","asserted_by":"not_stated","note_placement":"in_this_block","time_frame":"future_outcome","disclosure":"not_determined","horizon":"forward_looking"},{"claim_code":"MC-4032","claim_label":"Windermarsh Absolute Return strategy's modelled return","claim_kind":"hypothetical_figure","asserted_by":"firm","note_placement":"not_in_this_piece","time_frame":"period_ended","disclosure":"not_carried","horizon":"historical"},{"claim_code":"MC-2150","claim_label":"Kirtlow Diversified Fund ranked against its sector","claim_kind":"superlative","asserted_by":"third_party","note_placement":"elsewhere_in_piece","time_frame":"no_period","disclosure":"not_required","horizon":"not_determined"},{"claim_code":"MC-8629","claim_label":"Rossmere Balanced Fund's stated return","claim_kind":"performance_figure","asserted_by":"third_party","note_placement":"in_this_block","time_frame":"period_ended","disclosure":"not_required","horizon":"historical"},{"claim_code":"MC-7606","claim_label":"What Fennimore Balanced Fund is said to deliver next","claim_kind":"forward_statement","asserted_by":"not_stated","note_placement":"not_in_this_piece","time_frame":"future_outcome","disclosure":"not_determined","horizon":"forward_looking"}]}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Check fund marketing claims for missing disclosures — 50 marketing pieces. One model answered, and every answer was then graded Five different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran these runs publishes no rate card this repo commits, so nothing here is what was actually paid — the real spend for this kit's build is recorded in the commit history and the shared call ledger, not on this page.

The gradersFive ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The scan-line floor (evals/baseline.py), the scorer (evals/judge.py), the gold gate (evals/check_labels.py) and the stub all cost $0.00 -- no key, no provider, no tokens. The paid work behind this page is 50 scored calls, 6 calibration calls, 1 ceiling probe, 4 prompt-split calls at max_tokens=1, 1 worked-example replay and 1 screenshot call: 63 in total.

Cost driversWhat actually moves the bill

• THE OUTPUT, and it is not close. 484,164 output tokens against 251,193 input, and at a 6x rate differential that makes output 92 pct of the bill. 459,461 of those output tokens -- 94.9 pct -- are a provider-side reasoning pass rather than the worksheet.
• The fixed prefix -- system prompt (1,079 tokens), review checklist (1,771) and field schema (1,007) -- is 3,857 of every call's 4,920 input tokens, byte-identical on all 50 calls and paid again on every one. No caching is used and none is claimed.
• The piece itself is the SMALLEST part of the input at 1,063 tokens on the worked example. A kit whose answer is a lookup over a shipped rule pays more to carry the rule than to carry the document.

Your volumeWhat it costs at your volume

Linear in marketing pieces: each call is independent, carries the same fixed prefix and one piece, and nothing is shared or cached between them. 500 pieces is 10x the tokens and 10x the dollars. What is NOT linear is the output per piece -- 94.9 pct of it is reasoning whose length tracks how hard a piece is rather than how long it is, which this kit measured directly: a seven-claim piece came back cheaper than the most expensive six-claim one.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 50 marketing pieces, one call each, 275 claims and eight fields per claim, plus the piece reference.

The codeEvery file, and what it is for

Six modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pysegment

cut the marketing piece into addressable sections, pure code -- one per coded claim, plus the schedule, the performance table, the footer and any decoy section. ON THIS KIT A SECTION BOUNDARY IS PART OF THE ANSWER: the hardest field asks which BLOCK a note sits in, so 'this claim's own block' has to be a thing that exists

# Cut a marketing piece into addressable sections. Pure code — no model, no network.

def sections(text):
def locate(text, value):
def span_label(secs, start):

src/select.pyselect — a swap seam

pick which sections are sent, pure code -- names matched by PREFIX because every piece's claim blocks are named after its own claim codes. The three decoy sections are deliberately SENT; the Distribution Record is mapped by nothing and never leaves the machine

You change it to: map fields to your own piece's headings, including the per-claim prefix; unmatched falls back to the whole document. NEVER_SENT is the list of sections no field asks for and which therefore never leave the machine -- and evals/check_labels.py convicted a one-word mismatch here on this kit's first run, before a call was paid for

# Pick which sections of a marketing piece are sent. Pure code -- the last deterministic step

PIECE = "Piece"
SCHEDULE = "Schedule Of Claims"
PERFORMANCE = "Performance Summary"
IMPORTANT = "Important Information"
REVIEW_NOTES = "Review Notes"
SPECIMEN = "House Style Specimen"
PREVIOUS = "Carried From The Previous Version"
CLAIM_PREFIX = "Claim "
NEVER_SENT = ("Distribution Record",)

src/rulebook.pychecklist

the two worksheet calls, loaded from data/rulebook.json -- a seven-step disclosure rule with a stopping order and a three-way horizon rule. Data, not code, so a forker can open it and disagree with it, which on this kit is the whole point: what ships is invented here and is not any regulator's rule

# THE REVIEW CHECKLIST, loaded from data/rulebook.json. Pure code, no model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RULEBOOK_PATH = os.path.join(HERE, "data", "rulebook.json")
KINDS = ("performance_figure", "hypothetical_figure", "forward_statement", "superlative",
ASSERTERS = ("firm", "third_party", "not_stated")
PLACEMENTS = ("in_this_block", "elsewhere_in_piece", "not_in_this_piece")
TIMEFRAMES = ("period_ended", "future_outcome", "no_period")
DISCLOSURES = ("carried", "not_carried", "not_required", "not_determined")
HORIZONS = ("historical", "forward_looking", "not_determined")
def load(path=RULEBOOK_PATH):

src/prompt.pyprompt

assemble one call per piece for every claim, with the checklist RENDERED into it from data/rulebook.json rather than restated in prose -- a rule the model has never been shown is not a rule. One call per PIECE is forced rather than chosen: a per-claim call could not answer where in the whole piece a note sits

# Assemble the review-worksheet prompt. ONE prompt per marketing piece, every claim in the reply.

SYSTEM = (
RULEBOOK_TEXT = rulebook_block()
def field_schema(piece_fields, claim_fields):
def build(doc_text, secs, fields, selector):
def parse(raw, fields):

src/extract.pyextract — a swap seam

the AI layer, one provider one key -- plus the pure-code business-condition check downstream: a piece making a claim about a future outcome whose required note is nowhere in it at all goes back to the author

You change it to: the review-flag condition -- this kit ships one narrow rule (a forward-looking claim whose required note is nowhere in the piece) chosen over four broader candidates on a measured firing rate. A real review weighs the audience, the campaign and the fund's own documents

# Extract one marketing piece's review worksheet: segment, select, prompt, one model call, then a

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 32000
def load_fields():
def claim_fields(fields=None):
def piece_fields(fields=None):
def load_doc(piece_id):
def documents():

evals/judge.pyjudge

score identification, the extracted cells, the two calls, the two directions of a disclosure error and the determinacy split, all separately, pure code -- plus a no-gold consistency diagnostic

# Score a marketing-review worksheet run. PURE CODE -- gold is exact and every answer is one value

def norm(v):
def code_norm(v):
def equal(got, want):
def _rows_of(rec):
def score_identification(records, golds):
CELL_FIELDS = ("claim_label", "claim_kind", "asserted_by", "note_placement", "time_frame",
def score(fields, records, golds):
CALLS = {"disclosure": RB.DISCLOSURES, "horizon": RB.HORIZONS}
CONFIDENT = {"carried", "not_carried", "not_required", "historical", "forward_looking"}

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 5023 input and 9683 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's marketing pieces are entirely synthetic (tools/build_corpus.py, seed 20260822): no real firm, fund, strategy, index, benchmark, ranking body, data provider, product, campaign or person exists in the corpus, and nothing was fetched from anywhere. No regulator, supervisor, trade body or firm is named anywhere in the kit, and no rule of any kind is reproduced. The only outbound traffic the kit makes is one chat-completion request per piece to the configured provider, carrying the mapped sections of one piece plus the fixed prompt and the shipped checklist -- and the Distribution Record, which is the one part of a real proof a marketing team would most mind sending anywhere, is mapped by no field and never leaves the machine. Nothing is written outside the kit directory, there is no database, no auth and no multi-tenancy, and the local UI binds 127.0.0.1 only.

Read from the shared .env or the real environment only, never committed, never logged and never rendered. .env is gitignored from the first commit. On an adapter error the app redacts the key and base URL out of the message before returning it, so a misconfigured endpoint cannot echo a credential into a browser. tools/shoot_ui.mjs refuses to start if anything is already listening on its port, because a server it did not start may hold a real key and clicking would spend.

The experimentWe did not attack it — and four of five boundaries hold

Four boundaries hold. Two were confirmed by reading the code: no code path approves, rejects, clears or files anything, and no sentence in a piece can change what the pure-code guardrail does with the values it is given. Two are enforced by assertions that run before every spend: the Distribution Record is never sent, and every decoy-bearing section IS sent, on every piece that has one. The fifth -- whether text written to talk a model out of the checklist would move a call -- is UNMEASURED, and no attack was fired. ⚠︎ AND THE FOURTH BOUNDARY HAS ALREADY FAILED ONCE IN THIS KIT'S LIFE: a one-word mismatch between a section's name in the generator and in src/select.py made selection fall back to the whole document, and the approver's name, the mailing list and a sales contact's direct line went to the model on every piece lacking a decoy section. evals/check_labels.py convicted it by name on its first run, before a call was paid for. That is what a boundary holding on evidence looks like, and it is why the fallback was left able to do it rather than being silently patched around the detector. Confirmed by reading the code and by pre-flight assertions, not by an attack run. No results/redteam-*.json exists for this kit, on 2026-08-22 -- no attack run was fired; see redteam.why.

Four of the five boundaries hold, two of them confirmed by reading the code and two by assertions that run before every spend. The fifth is UNMEASURED and is named as such. The fourth is the one that has already failed once in this kit's short life, and it was the gate -- not a review, and not a run -- that caught it.

The result0 attack trials, and four of five boundaries checked here hold on evidence rather than on assertion -- including the one that already failed once and was caught by the gold gate before a call was paid for.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260822) and contains no adversarial text at all. The 39 decoy codes are confusable PLAIN copy, not attacks: a claim withdrawn by the review notes, a house-style specimen and a claim carried from the previous version. The run left all 39 off the worksheet. That is a resistance number about DRAFTING, not about injection, and it is published as one.

HonestyWhat this does not prove

• Whether an indirect prompt injection in a footer or a claim block could move a call. No attack was fired and the corpus contains none.
• Whether the disclaimer works. Six surfaces say the checklist is invented and that no piece is approvable by this kit, and nobody has tested whether a reader who skims takes any of it in.
• Whether a real proof's internal metadata is as separable as this corpus's Distribution Record. Here it is one named section; on real artwork it may be interleaved with the copy.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

needs_author_review fires when the piece makes at least one claim about a FUTURE OUTCOME whose required note appears NOWHERE IN THE PIECE AT ALL -- not beside the claim, not in the footer, not under the table, not beside another claim. It is a routing signal for a person, never a rejection and never an approval.

src/extract.py::compute(), called by extract() on every piece, by the local UI on every click, and by evals/judge.py over gold's own claims so the two are compared by the same function. Pure code, run on whatever the model returned and never on gold.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT an approval, a rejection, a sign-off, a clearance for distribution or a filing. NO PIECE IS APPROVABLE BY THIS KIT.
• It is NOT any regulator's test, any supervisor's expectation, any trade body's code or any firm's review policy. The condition was invented for this kit.
• It is NOT a check on the model. It is a business condition about the PIECE, and it fires identically on a perfect reply and on a wrong one that happens to carry the same three values.
• It is NOT a statement that the flagged piece is non-compliant, or that an unflagged one is compliant. It says one narrow thing about one narrow shape of gap.

WatchedWhat is watched, and why that one

The kit measures 15 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 8 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (documents 50, extraction_cells 1975, failures 0, refusal_cells 0) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

4 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 2 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

extraction · with the model in the path — 3 runs. Columns here are only ever compared with each other.

not a time series No two of these 3 runs measured the same system — they differ on documents, extraction_cells, provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

extraction · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 7 chips that all say so.

DeviationsWhat deviated

0 breaches across 4 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the flag's evidence on any change to compute() in src/extract.py, and re-measure its firing rate over gold BEFORE changing the condition -- the choice between the shipped rule and the four rejected ones was made on that measurement and on nothing else. Re-run the gold gate (free) on any change to data/rulebook.json, tools/build_corpus.py, data/fields.json or src/select.py. Re-run evals/run.py (paid) on any change to the provider, the model or MAX_TOKENS.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt names nothing, and the file says why: a requirements file naming a package the code never touches costs a forker an install and teaches them the kit is bigger than it is. The model is reached over urllib rather than through any vendor SDK, so a forker runs this on whichever key they already hold.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph. The pipeline has one path per piece -- segment, select, one call, parse, span, one pure-code condition -- with no branch, no loop, no tool call and no agent. The reply being a LIST is the only structural complication, and it is handled by scoring identification separately from the calls made about what was identified.

The other sideWhat a framework costs you

• Everything is hand-rolled, so everything is yours to maintain -- the JSON extraction from a reply that may arrive fenced, the span search, the retry loop and the budget ledger are all in this repo and all yours to fix.
• SECTION_HINTS is a layout assumption, not a parser. Point this at pieces with different headings and selection silently falls back to the whole document: slower, more expensive, always correct -- and able to smuggle a section you meant to withhold, which is exactly what evals/check_labels.py caught on this kit's first run.
• No structured-output mode is used. The reply is parsed from text, and a provider that supports a JSON schema would remove a failure mode this kit handles by failing closed instead.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-marketing-review on the fast tier, 2026-08-22. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 3 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

2 runs not plotted. b000-marketing-review-scanline, t000-marketing-review-stub recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the rulebook is data read whole into the prompt.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT -- this repository's own licence. Every firm, fund, strategy, benchmark, ranking body, data provider, person, figure and piece reference is invented; nothing was fetched and no real marketing piece was read Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one linePer-(claim, field) exact match, light normalisation

Does the model's value for each of the seven per-claim fields match gold, after trimming whitespace and punctuation, plus one piece_ref cell per piece? Nothing in this corpus is nullable -- every field is a value the piece states or a call the checklist makes -- so a null anywhere is a miss, never a legitimate blank.

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model. The same function evals/run.py calls after every run, and the same logic the free floor is scored by -- a floor and a real run scored by two different scorers cannot be compared honestly.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, read back off the same values the piece states; evals/check_labels.py asserts every field is populated and in vocabulary on every row, that every gold call is its own checklist lookup, that every note placement agrees with the piece's own text in BOTH directions, that the section vocabulary is closed, and that every claim code reaches the model, before any run is allowed to spend.

How tight can the band be? There is no tolerance band -- it is exact match after trimming whitespace and punctuation, never a continuous score to round.

Cadence: Re-score on any change to data/rulebook.json, src/prompt.py, data/fields.json or tools/build_corpus.py -- the first three change what is asked, the fourth changes what is asked about. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineWhich claims reached the worksheet

Micro precision and recall over claim codes, with the two directions named: a MISSED claim (one this version makes, left off) and a PHANTOM claim (one on the worksheet that this version does not make). Phantoms are broken down by which decoy produced them -- a claim withdrawn by the review notes, a house-style specimen, a claim carried from the previous version, or a code that is nowhere in the piece.

MethodHow the test was run

evals/judge.py::score_identification, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The 275 claim codes each piece's Schedule Of Claims lists for THIS version, and the 39 decoy codes gold records separately. check_labels.py asserts every decoy code is in the piece, collides with no real claim, REACHES THE MODEL, and clears a floor of 8 per kind.

How tight can the band be? A set comparison over normalised codes. There is nothing to threshold.

Cadence: Re-score on any change to the decoy design in tools/build_corpus.py or to rule 1 of the system prompt.

The decisionWhen to reach for it

In one lineThe disclosure call, and the two directions of being wrong about it

Three questions at once, over the claims BOTH sides list. Exact: did the run land on the same value the checklist produces -- carried, not_carried, not_required or not_determined? And the two that are NEVER averaged together: WAVED THROUGH, a claim that does not have the note the checklist asks for recorded as carried, and FALSE GAP, a claim whose note is properly in place recorded as not_carried. The first sends a piece onward carrying a claim nobody will look at again; the second costs a person an hour. They cost differently and an accuracy figure hides which one you are making.

MethodHow the test was run

evals/judge.py::score_calls, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: Gold's disclosure value is src/rulebook.py::disclosure() run over the same three facts the generator wrote into the piece, and check_labels.py re-derives every one of the 275 before any run may spend. It was red-proven by flipping a call, by making a placement disagree with the file it labels, and by relabelling a third-party claim; each was convicted by name.

How tight can the band be? Exact match over a four-value closed set. Nothing is rounded.

Cadence: Re-score on any change to data/rulebook.json or to rules 3 to 6 of the system prompt.

The decisionWhen to reach for it

In one lineThe horizon call

historical / forward_looking / not_determined on the same population, decided by the PERIOD the claim is about rather than the tense it is written in. 22 claims in this corpus state an outcome in a period that has not happened, in the past or present tense.

MethodHow the test was run

evals/judge.py::score_calls, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/rulebook.py::horizon() over the time_frame the generator wrote into the piece; check_labels.py asserts every back-worded claim is gold forward_looking and that at least 15 exist.

How tight can the band be? Exact match over a three-value closed set.

Cadence: Re-score on any change to the FORWARD_WORDED_BACK templates or to rule 7 of the system prompt.

The decisionWhen to reach for it

In one lineneeds_author_review, against the same rule run over gold

Does the run's own worksheet raise needs_author_review on the same pieces the rule raises it on when run over GOLD's claims? It is a BUSINESS condition -- a piece making a claim about a future outcome whose required note is nowhere in the piece at all -- so it needs labels, unlike the consistency diagnostic. A reply that did not answer is counted as unanswered, never folded into 'no'.

MethodHow the test was run

evals/judge.py::score_flag, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/extract.py::compute() run over gold's own claims. The condition was chosen on a MEASURED firing rate: four broader candidates were counted over the 50 gold pieces first, at 92, 74, 60 and 56 pct, and rejected for being flags nobody reads. check_labels.py refuses the gold set if the shipped condition falls outside 10-60 pct.

How tight can the band be? A boolean against a boolean. Nothing is thresholded.

Cadence: Re-check the flag's evidence on any change to compute() in src/extract.py, and re-measure the firing rate over gold before changing the condition.

The decisionWhen to reach for it