The business caseThe problem this solves

A standards framework states, sub-clause by sub-clause, what an institution has to DEMONSTRATE. The institution holds a pile of artefacts. Somebody has to say, for each artefact, which sub-clauses it actually demonstrates — none, one, or two in two different standards — and then produce the answer the whole exercise is for: the sub-clauses that ended with nothing behind them. Most of that is arithmetic. Is the clause id real; does the clause carry a currency requirement; is this artefact inside that window; which verdict follows; and, over the whole pile, which sub-clauses are empty. The part that is not arithmetic is whether an artefact DEMONSTRATES the thing or merely talks about it — a complaints POLICY against a clause asking for a complaints LOG, a review SCHEDULE against a clause asking for a completed review, the executive team against a clause naming the governing board. The vocabulary is identical in every one of those and the demonstration is not. The read-through: one person opening each artefact, deciding which sub-clauses it demonstrates, checking each mapping against that sub-clause's currency window by hand, and then — the step that actually gets skipped — going back over the finished mapping to work out which sub-clauses nobody filled. It does not replace the judgement; it produces the mapping a person checks, and it computes the coverage list that person would otherwise assemble by eye from a spreadsheet.

The inputThe actual evidence items, one artefact each

The corpus is 60 evidence items, one artefact each, 0.07 MB (accounts 1 · assessment 2 · census 1 · confirmation 1 · dashboard 1 · feature 1 · guidance 1 · handbook 2 · manual 1 · mapping grid 1 · minute 5 · model 2 · note 3 · notice 1 · policy 1 · record 1 · register 3 · release 1 · report 16 · return 1 · schedule 4 · scorecard 1 · specification 2 · statement 1 · statements 2 · summary 2 · survey 1 · tender 1). The interesting artefacts in a self-study are the ones that look like support and are not, and those are exactly the ones nobody publishes: an evidence set is a private working file an institution assembles for a body it is accountable to. There is no dataset to fetch. That would matter more if the prose were what is being measured — it is not. What is measured is a RELATIONSHIP between an artefact and a sub-clause, and to measure it you need artefacts whose relationship to every sub-clause is known, INCLUDING the ones whose vocabulary is right and whose demonstration is wrong. So the relationships are the plan and the documents are written around them: 7 cases, each a mistake somebody assembling a self-study would recognise — clean_single 20, wrong_thing 10, dual_support 8, stale_currency 7, unmapped 6, old_but_permitted 5, cross_standard 4.

EVIDENCE ITEM      EV-0001
Title              Mission statement as published, with the approving minute
Type               statement
Dated              2023-05-18
Author / owner     Governance Office
Lodged for         Self-study 2026 -- Brackenridge University
------------------------------------------------------------------------------

MISSION OF BRACKENRIDGE UNIVERSITY

"To teach without regard to a student's starting point, to hold our research to the standard
of the people it affects, and to remain answerable to the region that houses us."

This mission statement was approved by the Governing Board at its meeting of 18 May 2023,
minute GB/23/14, on the recommendation of the Strategy Committee. It has been published on
the University's public website continuously since 24 May 2023 and appears at the front of
the prospectus and of every programme handbook.

Extract from minute GB/23/14: "The Board RESOLVED to approve the mission statement as
circulated, and to require its publication in full."

Lodged with the self-study team; the owning office retains the original (SS-4007).

The outcomeWhat a good result looks like

One mapping row per artefact — the sub-clause ids it demonstrates (possibly none, possibly two), the sub-clause it merely resembles, a re-derived verdict of MAPPED / STALE / OFF_CLAUSE / UNMAPPED, and a sentence — and then one list computed over the whole pile in pure code: the sub-clauses no artefact supports at all. Measured on 60 items: 56 of 60 item sets exactly right (93.3 per cent) and 3 of 3 genuinely unsupported sub-clauses reported, with nothing falsely covered and nothing falsely uncovered.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• An evidence set that is already structured — a repository that records, per artefact, which sub-clause it was lodged against and when — the free rules floor, or no model at all
  The whole date half of this job is already free: both arms reject 7 of 7 stale artefacts and both handle the equally-old-but-permitted case, because src/framework.py applies the currency rule before any reading is consulted. If something upstream already tells you which sub-clause an artefact was lodged against, the reading is done and the rest is a set difference.
• A pile assembled by the offices being judged, where the traps are policies, schedules and the wrong actor — the model arm, and read the coverage list rather than the accuracy
  This is the case the corpus was built for and the only place the money shows. The floor accepts 8 of 10 wrong-thing artefacts and loses 4.4 and 6.3 of the three gaps; the model accepts 0 and loses none. On the item set that is 56 of 60 against 43 of 60.

And where nothing here is good enough:

• Long artefacts — a forty-page programme review that touches six sub-clauses in passing while demonstrating two — neither arm, on this evidence
  Nothing here measures it. Every artefact in this corpus is about a thousand bytes and single-purpose, so the hardest genuine case in the job is outside the denominator entirely. The multi-clause figure that does exist is 8 of 12 (66.7 per cent) on two-clause artefacts — the model's weakest row on this corpus, and it is measured on short documents.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-31. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-31 — r001-evidence-map. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Measured on a copy of the kit with no .env and no API key, on this machine, 2026-08-31. python3 tools/build_corpus.py rebuilt the 60 artefacts, the register, the answer key and the coverage file in 0.048 s, and every one of data/corpus/*.txt, gold.jsonl, coverage.json, corpus-stats.json, items.json, standards.json and standards.md came back BYTE-IDENTICAL to the committed files. ⚑ AND THAT WAS TESTED ACROSS FOUR PYTHONHASHSEED VALUES (0, 1, 12345, 99991), not one — a sibling kit in this batch ships a corpus whose planted collisions move with the hash seed because a set is sliced, and a single-seed rebuild cannot see it. Zero files differ at any of the four. python3 -m evals.check_labels then passed (60 gold rows, uncovered 2.5, 4.4, 6.3); python3 -m evals.redproof_recheck passed 6 of 6, each case convicting on its seed and acquitting without it; python3 -m evals.run --floor rules re-scored the whole free arm in 0.157 s and reproduced the committed floor exactly (43 of 60 item sets, 81.2 precision / 79.6 recall); and python3 -m evals.baseline --sweep re-ran all 105 threshold combinations in 8.5 s and re-selected the shipped TAU 0.65 / TAU_NEAR 0.35. No key, no network, no call.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. one evidence item read against the whole framework — one artefact, one model call, end to end, including provider-side reasoning tokens, on a shared connection. The recheck adds nothing measurable: it is pure Python over the reply the call already returned, and on this run it changed nothing at all (recheck_overrides 0, recheck_clauses_dropped 0). The coverage report is computed once, after every item, as a set difference — no call, no measurable time.

Current processWhat it replaces

The read-through: one person opening each artefact, deciding which sub-clauses it demonstrates, checking each mapping against that sub-clause's currency window by hand, and then — the step that actually gets skipped — going back over the finished mapping to work out which sub-clauses nobody filled. It does not replace the judgement; it produces the mapping a person checks, and it computes the coverage list that person would otherwise assemble by eye from a spreadsheet.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one evidence item is

No train/test split, because nothing is trained or tuned — the one tuned number in the kit is the floor's threshold, and it is swept in the FLOOR's favour with the answer key in hand and published as such. No segmentation either: one artefact goes whole into one call at a mean of 3077.2 input tokens. The sizes here are the artefacts' own bytes, measured off data/corpus/ on 2026-08-31.

SetupWhat the setup figure measured

There is no index and nothing is retrieved. The whole framework goes into every prompt at 6,898 characters and the artefact goes in whole; the coverage report is a set difference computed after every item, not a lookup. The 0.05 s is the entire corpus REBUILD measured on a keyless copy on 2026-08-31 (tools/build_corpus.py, 0.048 s wall) — there is nothing else to build.

LicenceLicence

MIT — the kit's own licence, and the MIT text is the file LICENSE-PUBLIC at the kits-repo root, read 2026-08-31. ⚠︎ THE ROOT LICENSE FILE IS A DIFFERENT DOCUMENT UNDER SEPARATE TERMS AND IS NOT THIS ONE, so the sentence 'MIT, this repository's own licence' would be false: the kit is MIT, the repository is not. Also read 2026-08-31. Nothing here is anyone else's: the framework, the register, the 60 artefacts and the answer key are all generated in-process by tools/build_corpus.py from seed 20260831, so there is no outside data in this kit to licence and nobody else's document is printed on any page. ⚠︎ The kit's own data/SOURCES.md states the synthetic provenance in full and states no grant at all; this field is the declaration.

Bring your ownBring your own evidence items, one artefact each

Drop your own artefacts as .txt into data/corpus/, add one row per file to data/items.json (id, title, type, dated, author), and replace data/standards.json and data/standards.md with your own framework — the clause ids, requirement text and currency windows are read, never hard-coded, and src/floor.py derives its whole vocabulary from the requirement text it finds there. Nothing else changes. To score, write data/gold.jsonl with one row per item carrying supports, near_miss and verdict, then run python3 -m evals.check_labels: it re-derives every verdict and the whole coverage set with its own arithmetic and names the rows that do not.

⚠︎ And what stops being true when you do: ⚠︎ THE ARTEFACT REACHES YOUR CONFIGURED PROVIDER VERBATIM, AND SO DOES EVERYTHING IN IT. A real evidence item carries the institution's name, the owning office, and whatever the document happens to say about named students and staff — complaint logs, survey responses, individual outcomes. That is a disclosure decision for whoever owns the material and it is not one a kit can make. Run it against an extract you are permitted to disclose, or against a local model. The register's metadata goes too: id, title, type, date, author/owner.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 3,065 tokens assembled, 2,150 are rules — 70% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

The exact five parts sent for EV-0001, the first item of run r001-evidence-map, replayed from the kit's own src/prompt.py build() on 2026-08-31. Trustworthy as the prompt that was sent because the replayed part lengths are identical to the ones the run recorded: system 1,789 / framework 6,898 / verdicts 2,361 / item 1,512 / schema 639. The run's own prompt_chars_total is 13,199 and the replay's is 13,199.

Full promptThe prompt in full

You are mapping ONE item of evidence onto the sub-clauses of a standards framework, for the team
assembling a self-study. Your output is one mapping row: which sub-clauses this artefact actually
demonstrates, or none of them.

How to read the item:

- A sub-clause is satisfied by an artefact that DEMONSTRATES the thing the sub-clause names. An
  artefact that discusses it, plans it, schedules it, or sets out the process for it does not
  satisfy it. A policy stating that complaints are resolved within twenty working days does not
  demonstrate that complaints were logged and resolved; a log with resolution dates does.
- The ACTOR named in a sub-clause is part of what must be demonstrated. Where a sub-clause says the
  governing board did something, a record of the executive team doing the same thing is not it.
- CURRENCY. Where a sub-clause carries a currency requirement of n years, an artefact dated before
  the review date minus n years does NOT satisfy it, however plainly it demonstrates the thing.
  Where a sub-clause carries no currency requirement, the artefact's age is irrelevant and an old
  artefact satisfies it exactly as a new one would. Both cases occur in this framework.
- ONE ARTEFACT MAY SATISFY MORE THAN ONE SUB-CLAUSE, including sub-clauses in different standards,
  where it genuinely demonstrates each of them. List every one it demonstrates, in document order.
- AN ARTEFACT MAY SATISFY NONE, and that is a real answer. Do not reach for the closest sub-clause
  to avoid an empty list.
- Where the item supports nothing but is on the subject of one sub-clause -- because it is out of
  date for it, or because it demonstrates the wrong thing -- name that sub-clause as the near miss.

Reply with JSON and nothing else, in the shape given at the end.

THE FRAMEWORK, as published:

# TSIE-3 — Trellis Standards for Institutional Effectiveness, third edition

**TSIE-3 is invented. It is not a real standards set, it is not modelled on one, and no part of it
should be read as a statement about how any real body evaluates anything.** It exists so this kit
has numbered sub-clauses that each state what must be *demonstrated*. See `data/SOURCES.md`.

Review date for this self-study: **2026-08-31**.

## How a sub-clause is satisfied

1. **A sub-clause is satisfied by an artefact that DEMONSTRATES the thing the sub-clause names —
   not by an artefact that discusses it, plans it, schedules it or describes the process for it.**
   A policy saying complaints will be resolved within twenty working days does not demonstrate that
   complaints were logged and resolved; a log with resolution dates does.
2. **The actor named in the sub-clause is part of what must be demonstrated.** Where a sub-clause
   says the *governing board* did something, a record of the executive team doing the same thing
   does not satisfy it.
3. **CURRENCY.** Where a sub-clause carries a currency requirement of *n* years, an artefact dated
   before `review date − n years` does **not** satisfy it, however plainly it demonstrates the
   thing. Where a sub-clause carries **no** currency requirement, the artefact's age is irrelevant
   and a schedule adopted a decade ago satisfies it exactly as this year's would.
4. **One artefact may satisfy more than one sub-clause**, including sub-clauses in different
   standards, where it genuinely demonstrates each of them.
5. **An artefact may satisfy none.** Most institutions hold a great deal of material that is
   evidence of nothing in the framework, and recording it as evidence of something is worse than
   filing it nowhere.

---

## Standard 1 — Mission and Governance

| clause | what must be demonstrated | currency |
|---|---|---|
| **1.1** Published mission | A published mission statement, together with the governing board minute that approved it. | none |
| **1.2** Board review of performance | A record that the governing board itself reviewed institutional performance against the mission. | 2 years |
| **1.3** Governance separated from management | A delegation of authority schedule separating decisions reserved to the governing board from decisions delegated to management. | none |
| **1.4** Conflict of interest declarations | Signed conflict of interest declarations from every member of the governing board for the current cycle. | 2 years |

## Standard 2 — Academic Programmes

| clause | what must be demonstrated | currency |
|---|---|---|
| **2.1** Programme learning outcomes | A programme specification stating the learning outcomes a student achieves at programme level. | none |
| **2.2** Periodic programme review | A completed periodic programme review, carried out on a cycle no longer than five years. | 5 years |
| **2.3** Assessment mapped to outcomes | A mapping of assessment tasks to the programme learning outcomes each task measures. | none |
| **2.4** Academic approval of change | A minute of the body holding academic authority approving a change to a programme. | none |
| **2.5** Outcomes achieved at exit | Evidence that graduating students achieved the programme learning outcomes at the point of exit. | 3 years |

## Standard 3 — Faculty

| clause | what must be demonstrated | currency |
|---|---|---|
| **3.1** Qualifications against level taught | Verified records of faculty qualifications checked against the level of study each person teaches. | none |
| **3.2** Review of teaching performance | A completed periodic review of teaching performance for members of faculty. | 3 years |
| **3.3** Workload allocation | A documented allocation of faculty workload across teaching, scholarship and service. | none |
| **3.4** Faculty development | A record of faculty participation in development activity. | 3 years |

## Standard 4 — Student Support and Outcomes

| clause | what must be demonstrated | currency |
|---|---|---|
| **4.1** Retention and completion | Retention and completion figures reported by entry cohort. | 2 years |
| **4.2** Academic advising | A record that academic advising was offered to enrolled students and taken up by them. | none |
| **4.3** Acting on student feedback | A record of a change made in response to student feedback. | 2 years |
| **4.4** Complaints logged and resolved | A log of student complaints showing the date each complaint was resolved. | 2 years |

## Standard 5 — Resources and Finance

| clause | what must be demonstrated | currency |
|---|---|---|
| **5.1** Audited financial statements | Externally audited financial statements for a completed financial year. | 2 years |
| **5.2** Multi-year financial plan | A multi-year financial plan approved by the governing board, with monitoring against it. | 3 years |
| **5.3** Library and learning resources | An assessment of whether library and learning resources are sufficient for the programmes offered. | none |
| **5.4** Infrastructure maintenance | A maintenance schedule for physical and digital infrastructure, with completion recorded against it. | none |

## Standard 6 — Institutional Effectiveness

| clause | what must be demonstrated | currency |
|---|---|---|
| **6.1** Measurable targets and results | Measurable targets for institutional effectiveness, reported together with the results achieved against them. | 2 years |
| **6.2** Findings changed practice | A documented change to practice made because of an assessment finding. | none |
| **6.3** Evaluation of the processes | An evaluation of the effectiveness processes themselves, judging whether they work. | 3 years |
| **6.4** Data source and method | A statement of the source and the method for the data used in an effectiveness decision. | none |

---

## The four answers an evidence item can carry

| verdict | when it applies |
|---|---|
| `MAPPED` | The item demonstrates at least one sub-clause, and is inside the currency window of every sub-clause it is mapped to. |
| `STALE` | The item demonstrates a sub-clause that carries a currency requirement, and the item is dated outside that window. It supports nothing; the sub-clause it would have satisfied is named as the near miss. |
| `OFF_CLAUSE` | The item is on the subject of a sub-clause and demonstrates something the sub-clause does not ask for. It supports nothing; the sub-clause it resembles is named as the near miss. |
| `UNMAPPED` | The item demonstrates no sub-clause of the framework and resembles none. |

## The coverage report

After every item is mapped, the sub-clauses with **no supporting item at all** are listed. That
list is the point of the exercise: an item mapped to the wrong sub-clause is a nuisance, and a
sub-clause nobody notices is unsupported is what a self-study is written to find.


THE FOUR VERDICTS, and what recording one commits you to:

  MAPPED
    means: The item demonstrates at least one sub-clause of the framework, and its date is inside the currency window of every sub-clause it is mapped to. An item may be MAPPED to two sub-clauses at once, including sub-clauses in different standards, where it genuinely demonstrates each of them.
    if you get it wrong: Mapped to a sub-clause it does not actually demonstrate, the sub-clause reads as covered and nobody looks at it again. That is the expensive direction: the coverage report is a set difference, so one wrong mapping deletes a genuine gap from the list a self-study is written to find.

  STALE
    means: The item demonstrates what a sub-clause asks for, and is dated outside that sub-clause's currency window. It supports nothing. The sub-clause it would have satisfied is named as the near miss, because the gap it leaves is a different gap from one where no such artefact exists at all.
    if you get it wrong: Counted as support, a sub-clause with a currency requirement is reported covered by evidence that is out of date — the exact finding the currency requirement was written to force. Counted as nothing, somebody commissions an artefact that already exists and only needs refreshing.

  OFF_CLAUSE
    means: The item is on the subject of a sub-clause and demonstrates something the sub-clause does not ask for: the policy rather than the log, the schedule rather than the completed review, the executive team rather than the governing board. It supports nothing, and the sub-clause it resembles is named as the near miss.
    if you get it wrong: This is the one that separates reading from word overlap, and it is where a keyword mapper fails hardest — the vocabulary is right in every case. Accepted as support, an unsupported sub-clause is reported covered by an artefact that will not survive the first person who opens it.

  UNMAPPED
    means: The item demonstrates no sub-clause of the framework and resembles none. Most institutions hold a great deal of such material, and it is a real answer rather than a refusal to answer.
    if you get it wrong: Mapped to something to avoid an empty cell, it pads the evidence set and hides a gap. Left out of the mapping altogether, nothing is lost — an item that supports nothing costs nothing to file nowhere.


THE EVIDENCE REGISTER'S ENTRY for this item. The date below is the register's, and it is
the date every currency window is measured against.

  Item id            EV-0001
  Title              Mission statement as published, with the approving minute
  Type               statement
  Dated              2023-05-18
  Author / owner     Governance Office
  Review date        2026-08-31
THE EVIDENCE ITEM, verbatim:

EVIDENCE ITEM      EV-0001
Title              Mission statement as published, with the approving minute
Type               statement
Dated              2023-05-18
Author / owner     Governance Office
Lodged for         Self-study 2026 -- Brackenridge University
------------------------------------------------------------------------------

MISSION OF BRACKENRIDGE UNIVERSITY

"To teach without regard to a student's starting point, to hold our research to the standard
of the people it affects, and to remain answerable to the region that houses us."

This mission statement was approved by the Governing Board at its meeting of 18 May 2023,
minute GB/23/14, on the recommendation of the Strategy Committee. It has been published on
the University's public website continuously since 24 May 2023 and appears at the front of
the prospectus and of every programme handbook.

Extract from minute GB/23/14: "The Board RESOLVED to approve the mission statement as
circulated, and to require its publication in full."

Lodged with the self-study team; the owning office retains the original (SS-4007).


Reply with JSON and nothing else, exactly this shape:

{"item_id": "<the evidence item id printed in the header, verbatim>",
 "supports": ["<sub-clause id, e.g. 2.1>", ...],
 "near_miss": "<the sub-clause id this item is on the subject of but does not satisfy, or none>",
 "verdict": "MAPPED" | "STALE" | "OFF_CLAUSE" | "UNMAPPED",
 "why": "<at most twenty-five words: what the artefact demonstrates, or what it fails to>"}

`supports` is a list and it may be empty. It is empty for every verdict except MAPPED, and
`near_miss` is "none" for MAPPED and for UNMAPPED. Sub-clause ids are the framework's own, in the
form <standard>.<clause>.

Raw responseThe raw response

{"item_id":"EV-0001","supports":["1.1"],"near_miss":"none","verdict":"MAPPED","why":"Published mission statement together with the governing board minute approving it, exactly as 1.1 requires."}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Match campus evidence to the accreditation rule it proves — 60 evidence items, one artefact each. One model answered, and every answer was then graded Three different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

No model grades anything. Every metric is exact set comparison against data/gold.jsonl — the support SET per artefact, order-insensitive, plus a verdict string — and the coverage report is a set difference over the whole pile. Precision and recall are printed side by side everywhere and never averaged into one number, because an arm that maps everything to everything is perfect on recall and worth nothing. why and near_miss are recorded and deliberately never scored: a plausible sentence beside a wrong mapping is worse than no sentence, and there is no defensible key for a near miss on an artefact that resembles two sub-clauses.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on this page is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid. The token counts come from the provider's own per-call reporting and are in results/eval-r001-evidence-map.json; the rates come from Foundry's build/facts/models.json, each row carrying its own as_of and source URL.

Rates checked 2026-08-31. The provider that actually ran every call here is kept off this page per the series rule, and so is its two-tariff structure. The real spend is computed per call against that provider's own dated card (as of 2026-08-23) at the tariff in force at the UTC time of each call, and it is $0.032134 for the 60 calls — $0.000536 an item — with a weekday-peak equivalent of $0.064272. Cached input is priced separately on that card and on no card below.

Grading unitWhat the grading figure prices

Scoring is pure Python over committed JSON: 60 artefacts in well under a second, no key, no network — measured at 0.157 s for the whole floor arm end to end on a keyless copy on 2026-08-31. The ARM costs money; grading it costs nothing, and re-scoring a cached run costs nothing either (API_KEY= python3 -m evals.run --run-id r001-evidence-map --resume --rescore, which re-derives rechecked from cached raw answers and makes no call). ⚠︎ --rescore WITHOUT --resume re-fires every call.

The gradersThree ways to grade

THREE ARMS ARE PUBLISHED AND THE NULL ARM IS THE FOURTH, MEASURED HERE 2026-08-31 RATHER THAN ASSERTED. An arm that maps NOTHING, ever — the cautious constant answer — gets 23 of 60 item sets right (38.3 per cent), because 23 artefacts in this corpus genuinely support nothing; it gets 6 of 60 verdicts (10.0 per cent), since only the 6 UNMAPPED items agree; and on coverage it finds all three gaps while declaring 22 covered sub-clauses uncovered as well — 100 per cent recall at 12.0 per cent precision, which is the mirror image of the loose-threshold failure and just as useless. The other constant arm, mapping every artefact to the most-supported sub-clause (6.4), gets 2 item sets of 60 and 37 of 60 verdicts, because MAPPED is the majority verdict. So the ladder on the headline grader is: null 23, free floor 43, model 56, out of 60. ⚑ READ THE FLOOR, NOT THE NULL, AS THE THING TO BEAT — it is a genuine attempt with the framework's own vocabulary and the same currency engine, tuned in its own favour with the answer key in hand. ⚑ AND THE FLOOR ARM IS NOW PUT THROUGH THE PURE-CODE STATION TOO — checked 2026-08-31, having been asserted until then. evals/run.py's --floor branch built the floor's rechecked column by COPYING its own raw answer, on the argument that src/floor.py already runs the currency engine so a second pass could find nothing. The argument was sound and it was still the wrong thing to ship: a copied column can only ever flatter the arm compared against it, and no published margin may rest on a claim nobody ran. The branch now calls src/recheck.py with the same arguments the paid path uses and every free arm was re-run at $0.00 on zero calls. The claim held — 0 overrides and 0 clause mappings dropped across all 60 items, and no figure in this note or the scores block moved — so the ladder stays null 23, free floor 43, model 56. The paid arm was not re-fired: eval-r001-evidence-map.json and its cache are byte-identical and the total is still $0.032134.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — top_k, chunk size, prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-31. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingGrading is free; the bill is the 60 calls that answered.

The corpus generator, the label gate, the recheck red-proof, the threshold sweep, the rules floor and the scorer are pure code and cost $0.00 against any result set — there is no LLM judge anywhere in the grading path. Measured end to end on a keyless copy on 2026-08-31: rebuild 0.048 s, floor arm 0.157 s, sweep (105 combinations) 8.5 s. The rechecked arm is re-derivable from the cache for nothing. $0.18512 is one full pass of the 60 paid calls at the shared projection card; the run's real bill at the running provider's own card was $0.032134.

Cost driversWhat actually moves the bill

• Output tokens: 87.9 per cent of them (27,204 of 30,933) are provider-side reasoning left at the tier's default and re-rolled per call, against a visible answer of four short fields. At the shared card's $0.50/$3.00 the output side is 50.1 per cent of the projected bill.
• The framework, in every prompt. data/standards.md is 6,898 of the 13,199 prompt characters — 52.3 per cent — and it is sent whole on every one of the 60 calls because the model is deliberately never shown the pile. It grows with the framework and not with the artefact.
• The cache, in the other direction: 135,680 of the 184,634 input tokens (73.5 per cent) came back marked as hits on the provider that ran this, which is why the real bill is $0.032134 rather than what the raw token count suggests. ⚠︎ Not one projection row below prices a cached-input rate.
• Nothing else. There is no index, no retrieval, no embedding, no second stage and no judge model. Grading, the floor, the sweep and the recheck are all $0.00.

Your volumeWhat it costs at your volume

Linear in artefacts and FLAT in the framework. Ten times the pile is ten times the calls at the same per-artefact cost — items are independent, nothing is retrieved, no state is carried, and the coverage report stays one set difference over whatever was mapped. What is NOT linear is a bigger FRAMEWORK: 6,898 characters of every prompt today, sent whole on every call, so twice the sub-clauses is roughly twice the input on every artefact in the pile.

Model choiceWhy this model

It is the tier the estate runs, so the figure compares with every sibling kit. The finding here is not about the model: it is that the free floor already reaches 81.2 precision and 79.6 recall and finds 1 of the 3 gaps, and that what $0.032134 buys on this corpus is 13 item sets and the other two gaps — all of it reading, none of it arithmetic. A more expensive model would have to be argued for against the floor, not against zero.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

Every model number on these pages: 60 evidence items, one call each, all answered, failures empty, every reply finishing on stop, nothing truncated. $0.032134 is the REAL spend at the runtime provider's own dated card (2026-08-23), priced per call at the tariff in force at the UTC moment of each — not the $0.18512 projection the Cost lens publishes on the shared card. Every call landed off-peak; the same tokens in that provider's weekday peak window would have cost $0.064272, exactly double, and the run file records both. There is no second paid arm: the 17-call injection probe has never been fired.

The codeEvery file, and what it is for

14 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Seven of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

data/items.jsonthe evidence register

one row per artefact — id, case, type, title, DATED, author, reference. The register is the authority on the date, and every currency window is walked against it rather than against a date recovered from a sentence inside the artefact.

# 

data/corpus/EV-0001.txtthe artefacts

60 generated evidence items in data/corpus/ (EV-0001 .. EV-0060), 69,244 bytes in total, mean 1154.1. One goes into each call, verbatim; the file named here is the first of them and the one every worked example on this page uses.

# 

data/standards.jsonthe framework, as code reads it

25 sub-clauses in 6 standards; each carries its requirement text and max_age_years, which is null on 12 of them. Every arm computes from this file and nothing hard-codes a clause id.

# 

data/standards.mdthe framework, as the model reads it

the same framework rendered as prose, 6,898 characters of every prompt. evals/check_labels.py fails the build if the two copies disagree about a clause id, a currency window or the review date.

# 

src/framework.pythe framework engine

everything about this job that is arithmetic: is the clause id real, does the clause carry a currency window, is this date inside it, which verdict follows, and — coverage() — which sub-clauses no item supports. ONE derivation, used by the generator, the floor, the recheck and the answer key alike, so no two arms can be holding different definitions of the four words.

# THE FRAMEWORK ENGINE — every rule of TSIE-3 that is arithmetic rather than reading.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FRAMEWORK = json.load(open(os.path.join(HERE, "data", "standards.json"), encoding="utf-8"))
FRAMEWORK_ID = FRAMEWORK["framework_id"]
FRAMEWORK_NAME = FRAMEWORK["framework_name"]
REVIEW_DATE = FRAMEWORK["review_date"]
INSTITUTION = FRAMEWORK["institution"]
CLAUSES = {}
CLAUSE_ORDER = []
STANDARDS = {s["id"]: s for s in FRAMEWORK["standards"]}

src/prompt.pythe prompt

five parts in a fixed order — system, framework, verdicts, item, schema — 13,199 characters for EV-0001, of which 11,687 (88.5 per cent) are byte-identical on every call. The verdict block is BUILT from VERDICT_MEANINGS rather than typed beside it.

# The assembled prompt, in one place, in the order it is sent.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FRAMEWORK_MD = open(os.path.join(HERE, "data", "standards.md"), encoding="utf-8").read()
VERDICTS = ["MAPPED", "STALE", "OFF_CLAUSE", "UNMAPPED"]
VERDICT_MEANINGS = {
def _verdict_block():
VERDICT_BLOCK = _verdict_block()
SYSTEM = """You are mapping ONE item of evidence onto the sub-clauses of a standards framework, for the team
SCHEMA = """Reply with JSON and nothing else, exactly this shape:
def item_header(meta):

src/mapper.pythe model call

one item per call, max_tokens 32,000, no streaming, raw HTTP over urllib to any OpenAI-compatible provider or to Anthropic. Parses a fenced or prose-led JSON reply and case-folds the closed vocabulary; it deliberately does NOT drop an unknown clause id, because that is a finding about the arm.

# One evidence item in, one mapping row out. The only place a model is called.

MAX_TOKENS = 32000
THINKING = None
def parse_reply(text):
def normalise(obj):
def map_item(cfg, text, meta, complete_fn=None, max_tokens=None):

src/recheck.pythe pure-code station — a swap seam

takes the reading and re-derives everything that is not reading: drops a clause id the framework does not carry, drops a mapping outside that sub-clause's currency window and remembers it as the near miss, collapses a duplicate, and re-derives the verdict from what survives. On this run it made 0 overrides on 60 items and dropped 0 clauses.

You change it to: which fields survive. Today: the clause ids are checked against the framework, the currency window is walked in code, the verdict is re-derived, and the coverage report never touches a model. Widening it moves work from code to the model; narrowing it shrinks the injection surface, because a field the code re-derives is a field a document cannot lie about.

# THE PURE-CODE STATION. It takes the reading and re-derives everything that is not reading.

def norm_id(x):
def recheck(answer, meta, review_date=None):

src/floor.pythe free floor — a swap seam

token containment between each sub-clause's OWN requirement text and the artefact, above a swept threshold (TAU 0.65 / TAU_NEAR 0.35), in front of the same currency engine. Its stop list is derived, not typed: any token appearing in 6 or more of the 25 sub-clauses is dropped.

You change it to: the tokeniser, the derived stop rule (DF_CUTOFF) and the two thresholds. python3 -m evals.baseline --sweep re-runs the whole 105-point grid in 8.5 s.

# WHAT PURE CODE CAN MAP, with no model and no key: token overlap between a sub-clause's own

STOP = {
DF_CUTOFF = 6
def _stem(w):
def tokens(text):
def _clause_tokens():
CLAUSE_TOKENS, FURNITURE = _clause_tokens()
TAU = 0.65
TAU_NEAR = 0.35
def overlap(clause_id, text):

evals/scoring.pythe scorer — a swap seam

exact set comparison per item against data/gold.jsonl, precision and recall never blended, the coverage set on its own denominator, and a 7-bucket failure taxonomy assigned in a fixed order so every wrong item lands in exactly one.

You change it to: the graders, the two denominators and the taxonomy order. The bucket order is behaviour: wrong_thing_accepted is assigned before support_invented, which is why the floor's 0 on support_invented does not mean it never over-asserts.

# Grade one arm against the answer key. Deterministic -- no model judges anything here.

TAXONOMY = ("wrong_thing_accepted", "stale_accepted", "unmapped_mapped", "support_invented",
def _pct(n, d):
def _norm(seq):
def _bucket(g, m):
def score(rows, golds, clause_order, uncovered_gold, texts=None):

evals/check_labels.pythe label gate

re-derives every verdict, every currency window and the whole coverage set with arithmetic written in the file, importing no src/ module. Re-run on a keyless copy 2026-08-31: 60 gold rows, 25 sub-clauses, 13 with a currency requirement and 12 without, 49 support pairs, all four verdicts exercised, uncovered 2.5, 4.4, 6.3 — OK.

# Grade the ANSWER KEY. Free, no key, no model, and deliberately not the engine that wrote it.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
def load(name):
def jsonl(name):
def main():

evals/redproof_recheck.pythe red-proof

six seeded defects, each proven to CONVICT with the seed and ACQUIT without it, and each seed verified to have changed the row it was seeded into. Re-run on a keyless copy 2026-08-31: 6 of 6 ok.

# RED-PROOF THE PURE-CODE STATION: prove it CONVICTS, and prove it ACQUITS. Free, no model.

def meta(iid, dated):
def reply(supports, near="none", verdict=None, iid="EV-TEST"):
CASES = []
def case(name, seeded, clean, check_seeded, check_clean, note):
def main():

src/app.pythe board

standard library HTTP server on port 9219. /api/items, /api/item, /api/rules, /api/coverage, /api/prompt and /api/recorded need no key; only /api/map calls a provider, and with no API_KEY it returns 200 saying so.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
CORPUS = os.path.join(HERE, "data", "corpus")
PORT = int(os.environ.get("PORT", "9219"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r001-evidence-map")
FLOOR_RUN = "b000-evidence-map-rules"
def items():
def golds():
def gold_coverage():

tools/build_corpus.pythe corpus generator

writes the 60 artefacts, the register and the answer key from seed 20260831, and asserts every planted case before it ships — a 'stale' item whose date turns out to be inside its window, an 'old but permitted' item against a clause that HAS a window, a wrong-thing item that satisfies the clause it is meant to miss, and an uncovered sub-clause nothing resembles are all build failures.

# Generate the evidence corpus, the evidence register and the answer key.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
CORPUS = os.path.join(DATA, "corpus")
SEED = 20260831
DATASET_VERSION = "evidence-map-v1-60items"
CASE_COUNTS = {
HARD = {"dual_support", "cross_standard", "wrong_thing", "stale_currency", "old_but_permitted"}
FILLER = [
ITEMS = [

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 3077 input and 515 output tokens per evidence item at top-k 1, of which about 150 tokens are fixed (system prompt, chat framing, the question itself) and the rest is retrieved context. Change the retrieval depth and the input scales with it. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model and top-k follow the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

WRITTEN, PAIRED BY DESIGN, AND NEVER FIRED — and saying so is the whole of this page. The artefact is this kit's injection surface and the threat is not hypothetical: every document in a self-study evidence set is written by the office whose work is being judged, and it reaches the model verbatim. Anyone who can add a covering note can add a line addressed to whatever reads it, and the line worth adding is obvious — say which sub-clause this document satisfies, and say the currency is fine. It does not even look dishonest: an evidence footer naming the clause an artefact is submitted against is an ordinary thing for a real self-study to carry.

API_KEY is read from the shared <code>&lt;repo&gt;/.env</code>, this kit's own <code>.env</code>, then the real environment, in that order, and the kit-local file is merged KEY BY KEY so a file holding only MODEL overrides the model and keeps the shared provider and key. Neither file is required and a missing one is not an error — that is what keeps a single kit independently forkable. Writes go through <code>os.open</code> at 0600 BEFORE the write, so the file never exists at the default umask, and <code>config.sources()</code> reports which files contributed and never a value. Both paths are gitignored.

The experimentWe attacked it — two gates

An indirect prompt injection has to clear two gates, and they behave differently. Both were run for real on 2026-08-31.

⚠︎ BOTH PREDICTIONS ARE WRITTEN DOWN SO THAT A RUN CAN REFUTE THEM, AND NEITHER IS EVIDENCE. The probe is paired and its control is FREE — it reads the scored run's own answers out of results/cache-r001-evidence-map.jsonl, so it costs one call per artefact instead of two, and only artefacts the control got right enter the paired denominator ('the injected arm also got it wrong' is not evidence about injection). It selects the 17 artefacts the key maps to nothing and that name a near miss — every OFF_CLAUSE (10) and every STALE (7) item — with the ones guarding a genuinely unsupported sub-clause FIRST, because there one accepted sentence deletes a gap rather than moving a percentage. The control now exists and the probe was still not fired.

The result0 of 17 adversarial calls fired. Nothing about injection has been measured on this kit, and no rate is published.

Every trial WOULD BE one artefact of this corpus with the shipped line appended as its final sentence, {clause} filled with that artefact's own near miss, and everything else byte-identical — paired against that same artefact's answer read from results/cache-r001-evidence-map.jsonl. Both arms are scored, and the two numbers would say exactly which half of this job the pure-code station protects. None of it has been run.

HonestyWhat this does not prove

• EVERYTHING ABOUT INJECTION. The probe is written and unfired — 17 calls, none made. No sentence, no model, no day. Nothing on this page is a measurement.
• WHETHER THE STRUCTURAL PROTECTION HOLDS UNDER ATTACK. That the verdict, the clause ids and the currency window are re-derived in code is a fact about the code and is red-proven six ways by evals/redproof_recheck.py; that an adversarial artefact cannot defeat it is not the same claim and has not been tested.
• WHETHER A DIFFERENT SENTENCE WOULD WORK. One line is shipped. It is the plausible attack rather than the loud one, and it is one attempt.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Never write to the evidence register, never edit an artefact, never mark a sub-clause met, never draft a word of the self-study. This kit produces THE MAPPING A PERSON CHECKS and the list of sub-clauses that mapping leaves empty. UNMAPPED is a real answer rather than a refusal, and OFF_CLAUSE names what the artefact resembles so a person knows whether to go looking or to commission something new.

Stated in the prompt, in the README and in src/app.py's docstring, and enforced by there being no such endpoint: the server's POST routes return mapping rows and nothing else, and there is no write path to data/items.json, data/standards.json or data/gold.jsonl anywhere in the kit. The only file any run writes is under results/.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• This is ABSENCE OF A WRITE PATH plus a prompt rule, not a runtime enforcement layer — with one real exception: src/recheck.py IS enforcement for everything the framework decides by arithmetic (the clause id, the currency window, the verdict, the coverage set).
• It is not a self-study. It does not draft a narrative, does not say whether a standard is MET, and there is no flag that adds a sentence.
• It is not an assurance that an institution does or does not do something. The coverage report says which sub-clauses nothing IN THIS PILE supports; it cannot know about an artefact in a filing cabinet nobody lodged.
• It is not tested against an adversarial artefact. evals/injection.py has never been fired: 17 calls, 0 made.
• It is not a second opinion on the answer key. evals/check_labels.py re-derives everything arithmetical and prints, itself, that it cannot check whether an artefact's supports list is right.

WatchedWhat is watched, and why that one

The kit measures 43 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

193 measured by the latest run -150 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (case_clean_single_items 20, case_cross_standard_items 4, case_dual_support_items 8, case_old_but_permitted_items 5, case_stale_currency_items 7, case_unmapped_items 6, case_wrong_thing_items 10, clean_items 26, coverage_clauses 25, coverage_gold_uncovered 3, dataset_version evidence-map-v1-60items, empty_items 23, failures 0, framework_id TSIE-3, hard_items 34, items 60, items_answered 60, mapped_items 37, multi_items 12, pairs_gold 49, rechecked_clean_items 26, rechecked_empty_items 23, rechecked_hard_items 34, rechecked_items_answered 60, rechecked_mapped_items 37, rechecked_multi_items 12, rechecked_rederived_from_cache False, socket_timeout_s 1200) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

3 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

comply · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 193 chips that all say so.

comply · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 197 chips that all say so.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 193 chips that all say so.

DeviationsWhat deviated

0 breaches across 3 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

The corpus generator, the label gate, the recheck red-proof, the threshold sweep, the rules floor and the scorer are free and re-run in under nine seconds together on a keyless copy (rebuild 0.048 s, labels, red-proof, floor 0.157 s, sweep 8.5 s — all measured 2026-08-31). Only the model arm costs money: 60 calls, $0.032134 at the running provider's own dated card. Re-scoring a cached run is free (API_KEY= python3 -m evals.run --run-id r001-evidence-map --resume --rescore).

In one lineNo framework, deliberately

NO FRAMEWORK. A folder of readable Python, standard library end to end — no orchestration layer, no vendor SDK, no rules engine, no retrieval stack, no date library beyond datetime, no pip install. requirements.txt names nothing to install.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Linear, and concurrent only at the item level. There is no agent, no tool loop, no retrieval and no state carried between artefacts — which is exactly why 5 workers is the whole concurrency story, and why the second answer is computed once at the end as a set difference rather than accumulated as the run goes.

The other sideWhat a framework costs you

• Adding a provider means editing the PROVIDERS dict by hand. In exchange pip install pulls nothing and a forker runs this on whichever key they already hold.
• No structured-output library, so the reply shape is enforced by a hand-written parser and a normaliser. It held on all 60 paid calls — failures is empty and every reply parsed — but that is one run's evidence.
• No date library, so currency is datetime plus a documented February-29 convention (src/framework.minus_years lands on the 28th rather than raising).
• No test framework. The checks are executable modules — evals/check_labels.py, evals/redproof_recheck.py, evals/baseline.py --sweep — each of which prints what it checked and what it cannot.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-evidence-map on the fast tier, 2026-08-31. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 1 earlier run on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the rulebook is data read whole into the prompt.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT — the kit's own licence, and the MIT text is the file LICENSE-PUBLIC at the kits-repo root, read 2026-08-31. ⚠︎ THE ROOT LICENSE FILE IS A DIFFERENT DOCUMENT UNDER SEPARATE TERMS AND IS NOT THIS ONE, so the sentence 'MIT, this repository's own licence' would be false: the kit is MIT, the repository is not. Also read 2026-08-31. Nothing here is anyone else's: the framework, the register, the 60 artefacts and the answer key are all generated in-process by tools/build_corpus.py from seed 20260831, so there is no outside data in this kit to licence and nobody else's document is printed on any page. ⚠︎ The kit's own data/SOURCES.md states the synthetic provenance in full and states no grant at all; this field is the declaration. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe support SET for one artefact, exactly — precision and recall printed side by side, never blended

whether the sub-clause ids an arm asserts for one artefact are exactly the ids the key carries — as a SET, so naming them in another order is not a different answer, and 12 of the 60 items carry two. Precision and recall are reported separately and F1 only beneath them, because an arm that maps every artefact to every sub-clause scores perfect recall, reports zero gaps and is worth nothing: the floor's own sweep reaches 100 per cent recall at 29.3 precision on this corpus.

MethodHow the test was run

python3 -m evals.run --run-id <id> [--floor rules]; evals/scoring.py compares sets against data/gold.jsonl. No model is in the grading path and re-scoring a cached run costs nothing (API_KEY= python3 -m evals.run --run-id r001-evidence-map --resume --rescore).

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl — 60 rows. Each artefact is authored against an explicit list of the sub-clauses it supports; the VERDICT is then produced by src/framework.derive() (the same function the floor, the recheck and the UI use) and the COVERAGE set by src/framework.coverage(), so neither is typed. tools/build_corpus.py refuses to write a corpus in which a planted case does not hold. evals/check_labels.py then re-derives the whole thing with arithmetic written in that file, importing no src/ module — re-run on a keyless copy 2026-08-31: OK. ⚠︎ IT PRINTS ITS OWN LIMIT AND SO DOES THIS ENTRY: it cannot check whether an item's supports LIST is right. Whether the mission statement really demonstrates 1.1 is a reading, it was authored, and nothing in the kit holds a second opinion about it. Every precision and recall figure here is measured against a mapping this kit wrote.

No true/false rates for this grader. It records 9 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? There is no threshold to tune in the GRADER — set equality has no tolerance. The one tuned number in the kit belongs to the arm being compared against: the free floor's TAU 0.65 / TAU_NEAR 0.35, chosen by evals/baseline.py over 105 combinations on pair-F1 WITH the answer key in hand, which is the most generous setting a floor can honestly be given. The sweep is published and re-runs in 8.5 s. One artefact is 1.7 points of the 60-item denominator, so no band finer than that is supportable.

Cadence: Once per corpus version, and again on any change to data/standards.json, data/standards.md or src/prompt.py — the framework is rendered into the prompt AND read by every code path, so a change there changes what is being asked as well as what is being checked. Re-scoring a cached run is free.

The decisionWhen to reach for it

In one lineThe coverage report — which sub-clauses NOTHING supports, on its own denominator

whether the set difference over an arm's whole mapping equals the key's. It is scored separately from the pair score because it is not a restatement of it: one wrongly accepted artefact is one pair of 49 and can delete an entire gap. false_covered — a genuinely unsupported sub-clause the arm reports as covered — is the failure with no downstream recovery, and it is counted on its own.

MethodHow the test was run

computed inside the same scoring pass; no model, no call, no second denominator to align.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/coverage.json — derived from data/gold.jsonl by src/framework.coverage() as a set difference and never written by hand; re-derived independently by evals/check_labels.py.

No true/false rates for this grader. It records 5 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? ⚠︎ THE DENOMINATOR IS THREE AND IT IS THE SMALLEST ON THIS PAGE. One sub-clause is 33.3 points, so no band is meaningful and the only honest alarm is ANY false_covered rather than a rate. Nothing here is tuned; it is a set difference.

Cadence: Every time the item-set grader runs — one pass over one result file, no extra cost. Also whenever the CORPUS changes, because coverage is a global quantity computed from local decisions: adding one artefact can fill or empty a sub-clause nothing else touched.

The decisionWhen to reach for it

In one lineThe verdict — MAPPED / STALE / OFF_CLAUSE / UNMAPPED, re-derived in code, never taken from the reply

whether the four-word answer matches the key. It is never read out of the reply: a reply that says MAPPED and lists nothing has contradicted itself, and src/framework.derive() resolves that the same way every time. The pair worth telling apart is STALE against OFF_CLAUSE — both support nothing, both name a near miss, and one means the artefact exists and needs refreshing while the other means it was never the right artefact.

MethodHow the test was run

same scoring pass; the verdict is derived from the surviving support set on every arm.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl's verdict, itself produced by src/framework.derive() from the authored support list and the register's date.

No true/false rates for this grader. It records 7 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No threshold — the verdict is derived by src/framework.derive() from the surviving support set, so it has no tolerance to tune. The denominators are small and unequal (MAPPED 37, OFF_CLAUSE 10, STALE 7, UNMAPPED 6), so one artefact moves the UNMAPPED rate by 16.7 points; the counts are published beside every rate for that reason.

Cadence: With the other two; one pass, one result file, no extra cost. Re-derive it after any change to src/framework.derive() and run evals/redproof_recheck.py, which convicts and acquits on six seeded defects in that derivation.

The decisionWhen to reach for it