The business caseThe problem this solves

A monitoring report says what a model predicted and what actually happened, segment by segment, and somebody has to say which segments have degraded past the line. The hard part is not the arithmetic. It is that the line is not one line -- it moves with the model's materiality tier -- and that two gates sit in front of it: a metric the report does not carry is a gap in the report, and a segment with too few observations or an outcome window that has not closed is one the policy declines to judge however alarming its numbers are. 44 of the 294 segments in this corpus have a calibration gap past their tier's red band and are unassessable anyway. A worksheet that reads the numbers and skips the gates calls a breach on all 44. Somebody in a model risk function opening a quarterly monitoring pack and working down the segment table by hand: reading the predicted rate against the realised one, checking each against the band for the model's materiality tier, remembering that a segment under the observation floor cannot be called either way, noticing that the drift table lists a segment the performance table left out, and catching the footnote that restates a baseline three pages later.

The inputThe actual monitoring reports

The corpus is 50 monitoring reports, 0.27 MB (txt 50). Plain text, one format, invented rather than fetched. A real monitoring pack names a real institution, a real model, a real book of business and a real person who owns it, and both the firm and its supervisor consider all of it confidential -- there is no public corpus of (monitoring report, correct findings) pairs for the same reason there is no public corpus of loan books. Generating it also makes the label mechanical in a way this shape needs: the generator chooses the intended STATUS first, synthesises numbers that produce it, rounds them to the precision the report actually prints, and re-tests them through src/policy.py until the ROUNDED values give the intended status. Gold is then the policy applied to the text, not somebody's reading of a table.

Synthetic Record
----------------
EVERY FIGURE, MODEL, SEGMENT, PERSON AND REFERENCE IN THIS FILE IS INVENTED. It was generated
by tools/build_corpus.py from a fixed seed for the model-degrade use-case kit. No institution,
portfolio, model, model owner, validator, supervisor or customer is real, no number is drawn
from any real book of business, and nothing here is a monitoring report anybody has produced
or is held to. It reproduces no supervisory guidance, regulation, standard or model risk
policy. Do not read it as one.

Monitoring Report
-----------------
Report reference        MPR-2026H1-219
Model identifier        MDL-9683
Model name              Mortgage Default
Materiality             Tier 1
Reporting period        2026 H1
Prior period            2025 H2
Prepared by             the model owner's team
Reviewed by             Model Risk Review
Monitoring policy       MP-2026.2

Owner Contact
-------------
Model owner             T. Iwasaki
Employee reference      EMP-877970
Desk extension          x6581
Owner mailbox           t.iwasaki@internal.invalid
Model inventory record  https://inventory.internal.invalid/models/MDL-9683

Model Overview
--------------
The model estimates probability of default at 24 months, residential mortgage.
Development sample window ended 2023. Development-sample AUC 0.784, whole book.
The approved baseline AUC for each segment is shown in Segment Performance below
and is not the development-sample figure above.

Segment Performance
-------------------
Period 2026 H1. Rates are outcome rates over each segment's own population.

  Remortgage
    Observations                    9,864
    Predicted rate                  0.1271
    Realised rate (2026 H1)         0.1110
    Realised rate (2025 H2)         0.1314

The outcomeWhat a good result looks like

A worksheet row per segment the report monitors -- its observation count, the predicted and realised rates normalised to percent, this period's AUC against the approved baseline, the PSI and whether the outcome window has closed -- with three findings on each naming the status and the number it was tested against. Plus a pure-code escalation flag and, beside every status, the status the shipped policy gives the run's OWN numbers, so a wrong answer says whether the number was misread or the line misapplied.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• First-pass reading of an ongoing monitoring pack, for a second-line reviewer who will read every flagged row themselves. — the one tier measured here -- 2,211 of 2,212 values read, 288 of 294 segments found including all 26 the performance table omits, 56 of 56 red findings with 0 false, and 396 of 396 ungradable slots called correctly with no over-calls. One report of the 50 returned nothing usable and is counted as missed throughout.
  Against the free table-scan floor: 99.95 pct extraction against 85.21, 56 of 56 reds against 50 of 58, and 0 false reds against 41. The floor's precision on the class the kit exists to name is 0.55. ⚠︎ The floor read all 50 reports and this run read 49, so those columns are not over the same denominator.
• Catching the segments a pack does not let anybody judge -- too few observations, an outcome window still open, or a metric simply not produced this cycle. — the one tier measured here. 396 ungradable slots, 0 over-called; 89 slots where the outcome window is not matured and the drift rule is nonetheless assessable, 0 wrong. Its two status errors run the other way -- a segment the policy DOES judge, called unjudgeable, off one misread window.
  This is the half a regex cannot do and the half a careless reader skips. 44 segments in this corpus have a calibration gap past their tier's red band and are unassessable anyway.
• Deciding whether to let a model apply the threshold at all, or to have it read the numbers and apply the line in code. — read the numbers with the model and apply the policy in code. That is what src/policy.py does over the reply -- though on this run it scored 862 of 864, exactly what the model's own call scored, because the only error was a misread value that code inherits.
  The arithmetic costs nothing in code, it cannot drift, and it is the same function gold was written with -- so the only thing the model is trusted with is the part a regex cannot do. On r001 the one status the run got wrong was a gate skipped over numbers it had already read correctly; on r002 the two wrong statuses are a gate applied correctly to a number read wrongly. Code fixes the first and cannot touch the second.

At a glanceHow the whole thing runs

Run once, for real, on 2026-09-15. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-09-15 — r002-model-degrade. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Checked on a fresh checkout with API_KEY left blank: python -m src.app starts on port 8972, the 50-report picker populates, any report renders in full, the shipped policy renders from data/policy.json in its own panel, and Build worksheet returns a plain sentence saying nothing was called. python -m evals.check_labels, python -m evals.judge --self-test and python -m evals.run --baseline all run to completion with no key and no network. requirements.txt is empty on purpose -- there is nothing to install.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per monitoring report

Current processWhat it replaces

Somebody in a model risk function opening a quarterly monitoring pack and working down the segment table by hand: reading the predicted rate against the realised one, checking each against the band for the model's materiality tier, remembering that a segment under the observation floor cannot be called either way, noticing that the drift table lists a segment the performance table left out, and catching the footnote that restates a baseline three pages later.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 1 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one report section is

cut on underlined headings -- one section per block of the pack: the header, the owner contact, the model overview, the segment performance table, the input drift table, the override log, the back-test table, the data quality notes, the owner commentary and the closing notes, plus the synthetic-record banner. It falls back to a single whole-document section when it finds no headings at all, because a forker's pack will not carry these ones and one honest section beats a spurious carve-up.

SetupWhat the setup figure measured

There is no index. Preparation is segmentation only -- 50 reports cut into 550 sections in 0.0107 s of pure Python, measured over the whole corpus with no model and no network. Nothing is retrieved: every section a field maps to is sent, and the figure is here so a reader can see that the preparation cost of this shape is not a cost at all.

LicenceLicence

MIT -- this repository's own licence. Every institution, model, segment, owner, validator and figure is invented, and each of the 50 files carries a Synthetic Record banner on its first line saying so. No supervisory guidance, regulation, standard, industry framework or firm's model risk policy is reproduced, quoted or named anywhere in the corpus or the policy.

Bring your ownBring your own monitoring reports

REPLACE data/policy.json FIRST. It is this kit's own construction, it reproduces no supervisory guidance, regulation, standard or institution's model risk policy, and every status on every page is a lookup against it. Your bands, your tier floors, your view on whether drift needs matured outcomes -- all of it is that one file, and the corpus generator, the prompt and the judge all read it, so changing it moves all three together. Then point tools/build_corpus.py at your own packs or write your own data/gold.jsonl in the same shape: one JSON object per report with a segments list, each segment carrying the seven read values and a findings list of {rule, status}. Everything downstream -- the six graders, the free floor, the escalation flag, the UI -- reads that shape and nothing else. Then edit src/select.py's NEVER_SENT and SECTION_HINTS for your own headings; the guard that keeps a section off the wire is only as good as the names in that file.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 4,965 tokens assembled, 1,554 are contexts — 31% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured token-for-token via evals/prompt_tokens.py's nested-prefix method on MPM-0044 -- four calls at max_tokens=1, each carrying the previous prefix plus exactly one more part, with each part's size taken as the difference between two consecutive prompt_tokens counts the provider itself returned. Not characters, and not an estimate. results/tokens-p001-model-degrade.json.

Full promptThe prompt in full

You read an ONGOING MODEL PERFORMANCE MONITORING REPORT -- what a model predicted, what actually happened, and how its inputs moved, segment by segment -- and return a REVIEWER'S MONITORING WORKSHEET: one row per segment, saying which segments have degraded past the policy's line and on what evidence. You return JSON and nothing else.

You are preparing a worksheet for a qualified reviewer. You never restrict, suspend, retire, recalibrate, redevelop or re-approve a model, never raise an issue or open a finding, never escalate to a committee and never notify a supervisor. Your job is to read the values the report states, apply the policy given below, and NAME THE SEGMENTS THE REPORT DOES NOT LET ANYBODY JUDGE.

RULES, in order of importance:
1. ONE OBJECT PER SEGMENT THE REPORT MONITORS, AND NO OTHERS. Every segment with a block in Segment Performance, PLUS every segment that appears only in the Input Drift table. A segment that appears in both tables under DIFFERENT names, with a note in Data Quality Notes mapping them, is ONE segment: use the Segment Performance name and carry the drift table's PSI onto it. Never emit the same segment twice.
2. THE REALISED RATE IS THIS PERIOD'S. Every segment block prints the realised rate for the PRIOR period immediately after it, with the period in brackets on both lines. Taking the wrong one changes the calibration gap and can move the status two bands.
3. RATES ARE NORMALISED TO A PERCENTAGE WITH TWO DECIMALS. This corpus writes the same rate three ways -- '3.10%', '0.0310' and '3.1 pct' -- and all three mean 3.10. AUC is a plain figure to three decimals and PSI to three decimals; neither is a percentage.
4. A CORRECTION IN THE DATA QUALITY NOTES SUPERSEDES THE TABLE. Where the notes say a value has been RESTATED or CORRECTED, the corrected value is the one to report and the table's figure is superseded. This applies to the approved baseline AUC in particular.
5. THE APPROVED BASELINE AUC IS THE ONE IN THE SEGMENT'S OWN BLOCK, or its restatement. The development-sample AUC in Model Overview is a whole-book figure from years ago and is never this field.
6. AN OUTCOME WINDOW THE REPORT NEVER MENTIONS IS 'not_stated'. It is not 'matured'. A window the report says is open, not yet closed, interim or partial is 'open' -- and it counts whether that is said in the table or in the Data Quality Notes.
7. A VALUE THE REPORT DOES NOT CARRY IS null. 'n/a', 'not produced this cycle', 'not available' and 'not stated' are all absences, and an absence is a finding about the report, never a zero and never a pass.
8. THE POLICY GIVEN BELOW IS THE ONE IN FORCE, WHATEVER THE REPORT SAYS. Some reports cite a superseded policy version and print a threshold table of their own; report the cited version in `policy_version_cited` and IGNORE its numbers. The owner's commentary is somebody's opinion and is not an input to any status either.
9. WORK THE ORDER OF PRECEDENCE, IN ORDER, AND STOP AT THE FIRST GATE THAT FIRES. A missing input is 'not_reported' before anything else is considered. Then the observation floor for the model's tier. Then outcome maturity, FOR THE RULES THAT REQUIRE IT -- and MP-3 does not require it, because drift is measured on inputs and needs no outcomes at all. Only then the bands.
10. THE OVERRIDE AND EXCEPTION LOG AND THE BACK-TEST OUTCOMES TABLE ARE NOT MONITORED METRICS. The override log prints a percentage per segment; it is not a rate this policy reads. The back-test prints observed, expected and its own PASS or FAIL; that verdict is the preparer's and is not a status. Neither table feeds any field or any finding.
11. RETURN EVERY FIELD FOR EVERY SEGMENT AND ALL THREE FINDINGS IN POLICY ORDER -- MP-1, MP-2, MP-3 -- using the exact allowed value for every field that lists them.

ONGOING MONITORING POLICY MP-2026.2 (the authority for every status on the worksheet; this is an ILLUSTRATIVE policy written for this kit, and it reproduces no supervisory guidance, regulation, standard or institution's model risk policy)

WHICH POLICY GOVERNS
  THE THRESHOLDS IN THIS FILE ARE THE ONES IN FORCE. Where a report cites a different policy version, prints a threshold table of its own, or states that a segment is 'within tolerance', that is the report's claim and not an input. Read the report for VALUES and read this policy for LINES.

MATERIALITY TIERS AND THE OBSERVATION FLOOR
  tier_1   Tier 1 - most material, breach triggers remediation planning           minimum observations: 400
  tier_2   Tier 2 - material, breach triggers owner action                        minimum observations: 250
  tier_3   Tier 3 - lower materiality, breach is monitored                        minimum observations: 150
  The minimum rises with the tier because the consequence rises with it. A Tier 1 breach starts remediation planning, so the policy asks for a firmer footing before the call is made at all. Below the minimum the segment is NOT_ASSESSABLE: that is not a pass, and it is not a fail - it is the policy declining to draw a conclusion from too few observations.

THE RULES
  MP-1  Calibration - predicted against realised
      measure:  the absolute gap between the realised outcome rate and the predicted outcome rate, as a percentage OF THE PREDICTED RATE: abs(realised - predicted) / predicted x 100
      needs:    predicted_rate_pct, realised_rate_pct
      matured outcomes required: YES
      direction: either. Over-prediction and under-prediction both count - a model that predicts 3.0 pct and realises 1.2 pct is as far off the line as one that predicts 3.0 pct and realises 4.8 pct, and the sign is recorded separately from the status.
      bands (relative percentage points of the predicted rate):
          tier_1   amber at or above 15.0     red at or above 30.0
          tier_2   amber at or above 25.0     red at or above 45.0
          tier_3   amber at or above 35.0     red at or above 60.0
  MP-2  Discrimination - AUC against the approved baseline
      measure:  the RELATIVE decline of this period's AUC from the approved baseline AUC: (baseline - current) / baseline x 100. An AUC at or above the baseline is a decline of zero or less and is WITHIN.
      needs:    auc_current, auc_baseline
      matured outcomes required: YES
      direction: one-sided. Only a decline counts. An improvement is not a breach.
      bands (relative percent decline from the approved baseline):
          tier_1   amber at or above 3.0      red at or above 6.0
          tier_2   amber at or above 5.0      red at or above 10.0
          tier_3   amber at or above 7.0      red at or above 14.0
  MP-3  Input drift - population stability index
      measure:  the population stability index reported for the segment against the approved baseline distribution, taken as an ABSOLUTE value with no tier adjustment.
      needs:    psi
      matured outcomes required: NO
      PSI IS AN INPUT MEASURE AND NEEDS NO OUTCOMES AT ALL. It compares the distribution of the scored population against the baseline distribution, which is knowable the moment the population is scored. A segment whose outcome window is still open therefore has a perfectly assessable PSI and an unassessable calibration, in the same row of the same table. That asymmetry is deliberate and it is the single most commonly mis-applied line in this policy.
      direction: one-sided. Only an increase counts.
      bands (absolute PSI):
          tier_1   amber at or above 0.1      red at or above 0.25
          tier_2   amber at or above 0.1      red at or above 0.25
          tier_3   amber at or above 0.1      red at or above 0.25

THE STATUS VOCABULARY
  within           the measure was computed and it is below the amber band
  amber            the measure reached the amber band and did not reach red
  red              the measure reached the red band. THIS IS DEGRADATION BEYOND THRESHOLD, and it is what the worksheet exists to name
  not_reported     the report does not carry a value this rule needs. A monitoring gap, recorded as a finding about the REPORT
  not_assessable   the values are there and the policy declines to draw a conclusion from them - too few observations, or outcomes that have not matured

ORDER OF PRECEDENCE -- work through IN ORDER, stop at the first that fires
  1. NOT_REPORTED FIRST. If any value the rule needs is absent from the report - not printed, printed as n/a, or stated as not produced this cycle - the status is `not_reported` and nothing else is evaluated. An absent number is a gap in the report, never a pass.
  2. THEN THE POPULATION FLOOR. If the segment's observation count is below the tier's `min_population`, the status is `not_assessable`. AND IF THE REPORT STATES NO OBSERVATION COUNT AT ALL, the status is `not_assessable` TOO, not `not_reported`: the floor is a positive test, an unstated count does not pass it, and the population is not a value any rule's `needs` list asks for - it is the evidence the policy needs before it will draw a conclusion from the values it did get.
  3. THEN OUTCOME MATURITY, FOR THE RULES THAT NEED IT. If the rule's `requires_matured_outcomes` is true and the segment's outcome window is anything other than `matured` - open, or not stated at all - the status is `not_assessable`. An outcome window the report does not mention is NOT assumed matured.
  4. ONLY THEN THE BANDS. Compute the measure and compare it against the tier's band for the rule: at or above red -> `red`; at or above amber -> `amber`; otherwise `within`.

WHY `not_assessable` AND `not_reported` ARE REAL ANSWERS
  A monitoring worksheet that only says met-or-breached has two ways of being useless. It calls a breach on forty accounts, which is noise; and it hides the segments nobody can judge behind a column of green. `not_assessable` and `not_reported` are the two answers a reviewer most needs, because both of them are work: one is a segment that has to wait, and the other is a report that has to be sent back.

Return these:
- report_id (string) -- the monitoring report reference from the Monitoring Report section, verbatim
- model_id (string) -- the model identifier from the Monitoring Report section, verbatim
- model_tier (enum) one of: tier_1, tier_2, tier_3 -- the model's materiality tier as the report states it, normalised to tier_1 / tier_2 / tier_3. This decides which band column of the policy applies to EVERY segment in the report
- policy_version_cited (string) -- the monitoring policy version the REPORT cites, verbatim, or 'not_stated' when it cites none. This is a reading of the report and NOT the policy you apply -- you always apply the policy supplied above, whatever the report cites
- segments (array of objects) -- one object per segment the report monitors, each carrying:
    - segment (string) -- the segment's name as the Segment Performance table writes it, verbatim. THIS IS THE KEY -- one object per segment the report monitors. When a segment appears only in the Input Drift table and nowhere in Segment Performance, use the name that table gives it. When the two tables use DIFFERENT names for the same segment and a note maps them, use the Segment Performance name once and do not emit the segment twice
    - population (integer) -- the observation count for this segment THIS period -- the observations line in Segment Performance, or the obs figure in Input Drift for a segment that appears only there. Digits only, no thousands separator. null when the report states none, which is a state the policy has its own rule for and is not a `not_reported` finding
    - predicted_rate_pct (number) -- the PREDICTED outcome rate for this segment this period, expressed as a percentage with two decimals. The report writes rates three ways -- '3.10%', '0.0310' and '3.1 pct' -- and all three mean 3.10. null when the report does not state it, states it as n/a, or says it was not produced this cycle
    - realised_rate_pct (number) -- the REALISED outcome rate for this segment THIS PERIOD, as a percentage with two decimals. Every segment block also prints the realised rate for the PRIOR period; that is not this field. null when not stated, n/a, or not produced
    - auc_current (number) -- this period's AUC for this segment, three decimals. null when not stated, n/a or not produced
    - auc_baseline (number) -- the APPROVED BASELINE AUC for this segment, three decimals. Where the Data Quality Notes state that a baseline has been RESTATED or CORRECTED, the restated value is the approved baseline and the table figure is superseded. The development-sample AUC in Model Overview is a different thing and is never this field. null when no approved baseline is stated
    - psi (number) -- the population stability index reported for this segment in Input Drift, three decimals. null when the segment has no PSI line, or its PSI is n/a or not produced
    - outcome_window (enum) one of: matured, open, not_stated -- whether this segment's outcome window has closed. 'matured' when the report says so; 'open' when the report says the window is open, not yet closed, or that the realised figure is interim or partial -- in the table or in the Data Quality Notes; 'not_stated' when the report says nothing either way. An unmentioned window is 'not_stated' and is NEVER assumed matured
    - findings (array of exactly 3 objects, in policy order MP-1, MP-2, MP-3), each carrying:
        - rule (enum) one of: MP-1, MP-2, MP-3 -- the policy rule this finding is about
        - status (enum) one of: within, amber, red, not_reported, not_assessable -- the status the policy's order of precedence produces for this rule on this segment
        - basis (string) -- one short clause: the number you measured and the band you tested it against, or the name of the gate that fired before any band was reached

Return a JSON object with exactly these top-level keys: report_id, model_id, model_tier, policy_version_cited, segments
`segments` is an array. Return it empty only if the report monitors no segment at all.

MONITORING REPORT
-----------------
Synthetic Record
----------------
EVERY FIGURE, MODEL, SEGMENT, PERSON AND REFERENCE IN THIS FILE IS INVENTED. It was generated
by tools/build_corpus.py from a fixed seed for the model-degrade use-case kit. No institution,
portfolio, model, model owner, validator, supervisor or customer is real, no number is drawn
from any real book of business, and nothing here is a monitoring report anybody has produced
or is held to. It reproduces no supervisory guidance, regulation, standard or model risk
policy. Do not read it as one.

Monitoring Report
-----------------
Report reference        MPR-2025Q4-756
Model identifier        MDL-2378
Model name              Deposit Attrition
Materiality             Tier 2
Reporting period        2025 Q4
Prior period            2025 Q3
Prepared by             the model owner's team
Reviewed by             Model Validation, Second Line
Monitoring policy       MP-2026.2

Model Overview
--------------
The model estimates probability of balance attrition over 6 months.
Development sample window ended 2019. Development-sample AUC 0.846, whole book.
The approved baseline AUC for each segment is shown in Segment Performance below
and is not the development-sample figure above.

Segment Performance
-------------------
Period 2025 Q4. Rates are outcome rates over each segment's own population.

  Notice Account
    Observations                    840
    Predicted rate                  0.0304
    Realised rate (2025 Q4)         0.0406
    Realised rate (2025 Q3)         0.0424
    AUC, this period                0.728
    AUC, approved baseline          0.742
    Outcome window                  open, closes next cycle

  Fixed Term
    Observations                    5,185
    Predicted rate                  n/a
    Realised rate (2025 Q4)         0.0770
    Realised rate (2025 Q3)         0.0725
    AUC, this period                0.691
    AUC, approved baseline          0.688

  Youth Saver
    Observations                    336
    Predicted rate                  6.18%
    Realised rate (2025 Q4)         10.03%
    Realised rate (2025 Q3)         8.31%
    AUC, this period                0.628
    AUC, approved baseline          0.711

  Business Reserve
    Observations                    216
    Predicted rate                  12.33 pct
    Realised rate (2025 Q4)         1.26 pct
    Realised rate (2025 Q3)         0.95 pct
    AUC, this period                0.775
    AUC, approved baseline          0.945
    Outcome window                  matured

  Offset Linked
    Observations                    7,154
    Predicted rate                  4.50%
    Realised rate (2025 Q4)         4.41%
    Realised rate (2025 Q3)         4.46%
    AUC, this period                0.749
    AUC, approved baseline          0.740
    Outcome window                  matured

  Dormant Reactivated
    Observations                    4,772
    Predicted rate                  10.10%
    Realised rate (2025 Q4)         12.07%
    Realised rate (2025 Q3)         14.72%
    AUC, this period                n/a
    AUC, approved baseline          0.727
    Outcome window                  matured

  Wealth Cash Hub
    Observations                    not stated
    Predicted rate                  0.0393
    Realised rate (2025 Q4)         0.0736
    Realised rate (2025 Q3)         0.0700
    AUC, this period                0.544
    AUC, approved baseline          0.661

Input Drift
-----------
Population stability index against the approved baseline distribution.

  Notice Account                     obs 840        PSI 0.051
  Fixed Term                         obs 5,185      PSI 0.016
  Youth Saver                        obs 336        PSI 0.015
  Business Reserve                   obs 216        PSI 0.065
  Offset Linked                      obs 7,154      PSI 0.193
  Dormant Reactivated                obs 4,772      PSI 0.468
  WEALTH CASH HUB                    obs not stated PSI 0.173
  Instant Access                     obs 2,835      PSI 0.498

Override and Exception Log
--------------------------
Manual overrides of the model score, and exceptions raised by the owner's team.

  Notice Account                     overrides 149     of 840        17.74%   exceptions raised 4
  Fixed Term                         overrides 921     of 5,185      17.76%   exceptions raised 1
  Youth Saver                        overrides 32      of 336         9.52%   exceptions raised 2
  Business Reserve                   overrides 6       of 216         2.78%   exceptions raised 1
  Offset Linked                      overrides 987     of 7,154      13.80%   exceptions raised 2
  Dormant Reactivated                overrides 678     of 4,772      14.21%   exceptions raised 3

Back-Test Outcomes
------------------
Observed against expected outcomes, matured segments only. Binomial test at 95 pct.

  Business Reserve                   observed 3        expected 27       allowed 35       PASS
  Offset Linked                      observed 315      expected 322      allowed 383      PASS
  Dormant Reactivated                observed 576      expected 482      allowed 572      FAIL

Data Quality Notes
------------------
  - The outcome window for Fixed Term has not yet closed; the realised figure shown above is
    an interim reading and will be restated next cycle.
  - The approved baseline AUC for Business Reserve was RESTATED to 0.876 following the most
    recent revalidation. The figure of 0.945 shown in the table above is the superseded
    value and is retained only for continuity.
  - The segment shown as 'WEALTH CASH HUB' in the drift table is the segment reported as
    'Wealth Cash Hub' in Segment Performance. The two teams use different labels and no
    reconciliation has been agreed.
  - Instant Access is monitored for input drift only this cycle. No performance figures were
    produced for it because the outcome extract did not include the segment.

Owner Commentary
----------------
  The owner's assessment is that all segments remain within tolerance this
  cycle and that no threshold has been breached. Movements against the prior
  period are considered to reflect portfolio mix rather than model
  performance, and no remediation is proposed.

Report Notes
------------
  Prepared from the monitoring extract dated the last business day of the period.
  Figures are as produced by the reporting run and have not been re-derived here.
  Queries to the model owner's team.

Raw responseThe raw response

{
  "report_id": "MPR-2025Q4-756",
  "model_id": "MDL-2378",
  "model_tier": "tier_2",
  "policy_version_cited": "MP-2026.2",
  "segments": [
    {
      "segment": "Notice Account",
      "population": 840,
      "predicted_rate_pct": 3.04,
      "realised_rate_pct": 4.06,
      "auc_current": 0.728,
      "auc_baseline": 0.742,
      "psi": 0.051,
      "outcome_window": "open",
      "findings": [
        { "rule": "MP-1", "status": "not_assessable", "basis": "outcome window open" },
        { "rule": "MP-2", "status": "not_assessable", "basis": "outcome window open" },
        { "rule": "MP-3", "status": "within", "basis": "PSI 0.051 below amber 0.1" }
      ]
    },
    {
      "segment": "Fixed Term",
      "population": 5185,
      "predicted_rate_pct": null,
      "realised_rate_pct": 7.70,
      "auc_current": 0.691,
      "auc_baseline": 0.688,
      "psi": 0.016,
      "outcome_window": "open",
      "findings": [
        { "rule": "MP-1", "status": "not_reported", "basis": "predicted rate not reported" },
        { "rule": "MP-2", "status": "not_assessable", "basis": "outcome window open" },
        { "rule": "MP-3", "status": "within", "basis": "PSI 0.016 below amber 0.1" }
      ]
    },
    {
      "segment": "Youth Saver",
      "population": 336,
      "predicted_rate_pct": 6.18,
      "realised_rate_pct": 10.03,
      "auc_current": 0.628,
      "auc_baseline": 0.711,
      "psi": 0.015,
      "outcome_window": "not_stated",
      "findings": [
        { "rule": "MP-1", "status": "not_assessable", "basis": "outcome window not stated" },
        { "rule": "MP-2", "status": "not_assessable", "basis": "outcome window not stated" },
        { "rule": "MP-3", "status": "within", "basis": "PSI 0.015 below amber 0.1" }
      ]
    },
    {
      "segment": "Business Reserve",
      "population": 216,
      "predicted_rate_pct": 12.33,
      "realised_rate_pct": 1.26,
      "auc_current": 0.775,
      "auc_baseline": 0.876,
      "psi": 0.065,
      "outcome_window": "matured",
      "findings": [
        { "rule": "MP-1", "status": "not_assessable", "basis": "population 216 below tier_2 minimum 250" },
        { "rule": "MP-2", "status": "not_assessable", "basis": "population 216 below tier_2 minimum 250" },
        { "rule": "MP-3", "status": "not_assessable", "basis": "population 216 below tier_2 minimum 250" }
      ]
    },
    {
      "segment": "Offset Linked",
      "population": 7154,
      "predicted_rate_pct": 4.50,
      "realised_rate_pct": 4.41,
      "auc_current": 0.749,
      "auc_baseline": 0.740,
      "psi": 0.193,
      "outcome_window": "matured",
      "findings": [
        { "rule": "MP-1", "status": "within", "basis": "2.00% relative gap below amber 25.0%" },
        { "rule": "MP-2", "status": "within", "basis": "AUC 0.749 above baseline 0.740" },
        { "rule": "MP-3", "status": "amber", "basis": "PSI 0.193 at/above amber 0.1 below red 0.25" }
      ]
    },
    {
      "segment": "Dormant Reactivated",
      "population": 4772,
      "predicted_rate_pct": 10.10,
      "realised_rate_pct": 12.07,
      "auc_current": null,
      "auc_baseline": 0.727,
      "psi": 0.468,
      "outcome_window": "matured",
      "findings": [
        { "rule": "MP-1", "status": "within", "basis": "19.50% relative gap below amber 25.0%" },
        { "rule": "MP-2", "status": "not_reported", "basis": "AUC current not reported" },
        { "rule": "MP-3", "status": "red", "basis": "PSI 0.468 at/above red 0.25" }
      ]
    },
    {
      "segment": "Wealth Cash Hub",
      "population": null,
      "predicted_rate_pct": 3.93,
      "realised_rate_pct": 7.36,
      "auc_current": 0.544,
      "auc_baseline": 0.661,
      "psi": 0.173,
      "outcome_window": "not_stated",
      "findings": [
        { "rule": "MP-1", "status": "not_assessable", "basis": "observation count not stated" },
        { "rule": "MP-2", "status": "not_assessable", "basis": "observation count not stated" },
        { "rule": "MP-3", "status": "not_assessable", "basis": "observation count not stated" }
      ]
    },
    {
      "segment": "Instant Access",
      "population": 2835,
      "predicted_rate_pct": null,
      "realised_rate_pct": null,
      "auc_current": null,
      "auc_baseline": null,
      "psi": 0.498,
      "outcome_window": "not_stated",
      "findings": [
        { "rule": "MP-1", "status": "not_reported", "basis": "predicted and realised rates not reported" },
        { "rule": "MP-2", "status": "not_reported", "basis": "AUC not reported" },
        { "rule": "MP-3", "status": "red", "basis": "PSI 0.498 at/above red 0.25" }
      ]
    }
  ]
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch which of a lender's loan segments have degraded — 50 monitoring reports. One model answered, and every answer was then graded Six different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran this run publishes no rate card this repo commits, so nothing here is what was actually paid -- the real spend for this kit's build is 111 calls recorded in the shared call ledger -- 61 before r002 and 50 more for r002 itself, not on this page.

The gradersSix ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The free floor (evals/baseline.py), the scorer (evals/judge.py), its --self-test and the gold gate (evals/check_labels.py) are all pure code and cost $0.00. The figure beside this note is one full pass of the scored run on the projected card. The kit's whole build cost 111 calls: 50 scored on r001, 50 scored on r002, 6 calibration, 4 prompt-token probes at max_tokens=1, and 1 worked example.

Cost driversWhat actually moves the bill

• PROVIDER-SIDE REASONING, WHICH IS 70.1 PCT OF THE OUTPUT TOKENS (156,385 of 223,207). Output is 85 pct of the bill on the published card, so this one number is most of the cost of the kit.
• THE SEGMENT COUNT, which drives the reply length directly -- one object with three findings per segment. The longest reply on this run used 9,482 output tokens against a corpus average of 4,555.
• THE FIXED PREFIX, 3,411 of 4,965 input tokens on MPM-0044 -- the system prompt, the whole monitoring policy and the field schema, identical on every call. It is 15 pct of the bill at this card's rates, and it is the part that GROWS when a forker swaps in a real policy.

Your volumeWhat it costs at your volume

Linear in reports: each call is independent, carries the same fixed prefix and shares nothing with the others, so 500 reports is 10x the tokens and 10x the price. Wall time is not linear because the calls parallelise -- this run took 72.6 seconds for the 49 reports that returned, against 761 seconds of summed call time on 12 workers. What does grow super-linearly is a report with more segments, because the reply grows with the segment count and the output side is 85 pct of the bill.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 50 monitoring reports, one completion call each; 49 replies parsed and were scored, carrying 288 segments, seven read values on each and three policy findings on each, plus four report-level fields. The tokens below are billed for all 50 calls, including the one whose reply came back empty.

The codeEvery file, and what it is for

Six modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Six of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pysegment

cut the report into its eleven addressable sections, pure code -- the header, the owner contact block, the overview, the performance table, the drift table, the override log, the back-test table, the notes, the commentary and the closing prose. 550 sections across 50 reports in 0.0107 s. The cut is what lets a cell cite WHERE it was read from, which matters here because the same segment is named in four tables and two of them are decoys.

# Cut a monitoring report into addressable sections. Pure code -- no model, no network.

def sections(text):
def locate(text, value):
def span_label(secs, start):

src/select.pyselect — a swap seam

decide what is sent, pure code. Owner Contact is mapped by no field and never leaves the machine on any of the 50; the override log and the back-test table ARE sent, deliberately, because they are the decoys the run is being measured on.

You change it to: what reaches the provider and what never leaves the machine. NEVER_SENT is the list a second-line function will want to edit first.

# Pick which sections of a monitoring report are sent. Pure code -- the last deterministic step

BANNER = "Synthetic Record"
HEADER = "Monitoring Report"
CONTACT = "Owner Contact"
OVERVIEW = "Model Overview"
PERF = "Segment Performance"
DRIFT = "Input Drift"
OVERRIDES = "Override and Exception Log"
BACKTEST = "Back-Test Outcomes"
NOTES = "Data Quality Notes"

src/policy.pypolicy

the thresholds and the order of precedence, read as DATA from data/policy.json. One definition with three readers -- the corpus generator writes gold with it, the prompt states it to the model, and the judge re-runs it over the run's own numbers.

# THE MONITORING POLICY, loaded from data/policy.json. Pure code, no model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
POLICY_PATH = os.path.join(HERE, "data", "policy.json")
STATUSES = ("within", "amber", "red", "not_reported", "not_assessable")
TIERS = ("tier_1", "tier_2", "tier_3")
WINDOWS = ("matured", "open", "not_stated")
def load(path=POLICY_PATH):
P = load()
RULE_CODES = tuple(r["code"] for r in P["rules"])
RULES = {r["code"]: r for r in P["rules"]}

src/prompt.pyprompt

one call per report. The policy is rendered into every prompt from the JSON file rather than restated in prose -- 1,511 of 4,965 input tokens on MPM-0044 -- so the model's instructions and the gold labels cannot drift about the same threshold.

# Assemble the monitoring prompt. ONE prompt per report, every segment in the reply.

SYSTEM = (
POLICY_TEXT = policy_block()
def field_schema(report_fields, seg_fields, finding_fields):
def build(doc_text, secs, fields, selector):
def parse(raw, fields):

src/extract.pyextract — a swap seam

the single completion call, the span for every located value, the shipped policy re-applied IN CODE to whatever the model returned, and the escalation flag.

You change it to: the escalation condition. It is a BUSINESS rule, not a check on the model, and it was chosen against a measured firing table rather than picked because it sounded right.

# Assemble one monitoring report's worksheet: segment, select, prompt, one model call, then a

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 48000
def load_fields():
def segment_fields(fields=None):
def report_fields(fields=None):
def finding_fields(fields=None):
def load_doc(doc_id):

evals/judge.pyjudge

six graders, all pure code, plus a --self-test that red-proves each one against a deliberately broken record. No model, no key, no cost.

# Score a run against gold. PURE CODE -- no model, no key, no network, no cost.

CELL_FIELDS = ("population", "predicted_rate_pct", "realised_rate_pct",
REPORT_CELLS = ("report_id", "model_id", "model_tier", "policy_version_cited")
TOL = {"predicted_rate_pct": 0.005,
def norm_name(v):
def _num(v):
def cell_match(field, got, want):
def flat(row):
def score_roster(records, golds):
def score(fields, records, golds):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 4727 input and 4555 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's monitoring reports are entirely synthetic (tools/build_corpus.py, seed 20260822): no institution, model, portfolio, owner, validator, supervisor or customer in them is real, and each of the 50 files says so on its first line. Nothing is uploaded, nothing is written outside the kit directory, and the only network call in the kit is one completion per report to the configured provider.

Read from the shared .env or the real environment only, never committed, never logged and never rendered. The local app strips the key and the base URL out of any provider error before it reaches the page, and .env is gitignored from the first commit -- this repository has never held a credential and is not able to.

The experimentWe did not attack it — and one boundary was measured rather than argued

Four boundaries were examined and only one of them is an experiment. Two were confirmed by reading the code: no path writes, notifies or approves anything, and the Owner Contact block is mapped by no field. That second one is asserted over all 50 reports and RED-PROVEN -- evals/check_labels.py runs the selector with a hint that matches nothing at all, and shows the shipped fallback leaking on 0 of 50 where the naive one leaks on 50 of 50. The third was measured on the scored run: 7 reports cite a superseded policy and print looser thresholds in their own header, and all 7 came back with the cited version captured as a field and every status matching the SHIPPED policy. The fourth was not tested at all. Nobody wrote an instruction into the Owner Commentary and re-ran, so this kit publishes an attack count of zero rather than a resistance rate. Confirmed by reading the code, by a pre-flight assertion over all 50 reports that is red-proven against the naive fallback, and on one boundary by the scored run itself -- not by an attack, because none was fired. Checked 2026-09-15 -- no attack run was fired; see redteam.why.

Four boundaries, and the interesting thing is that they are not all the same strength. Two were confirmed by reading the code and asserting it over the whole corpus (nothing is written or notified; the owner contact block never leaves the machine, red-proven). One was MEASURED on the run (the superseded threshold table moved nothing on 7 of 7). One is UNTESTED and is published as untested: nobody attacked the two externally-authored sections, and 12 reports of ordinary misleading prose is evidence about prose, not about injection.

The result0 attack trials, and four of four boundaries examined here hold on evidence of three different strengths -- two read off the code and asserted over all 50 reports, one measured on the run (7 of 7 superseded threshold tables moved no status), and one that is honestly untested.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260822), so no text in it came from anywhere but this repository and there is no live injection surface to measure. The two sections that WOULD be an injection surface in a real deployment are sent on purpose, because hiding them would hide the kit's own attack surface from the run that is supposed to measure it -- and then nobody measured it. That is the honest state and it is what the headline says.

HonestyWhat this does not prove

• Whether a sentence written into the Owner Commentary or the Report Notes can move a status. Both sections are sent and no attack was fired; the 12 reports of misleading prose in this corpus are ordinary prose, not an injection test.
• Whether the withholding guard survives a forker's own headings. It keys on the literal section name Owner Contact; rename it in a real pack and the guard silently stops applying, which is the first thing a forker should check and the thing this corpus cannot check for them.
• Whether a real provider retains what is sent. The kit sends one completion request and reads the reply; what happens to the payload after that is the provider's policy and nothing here measures it.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

needs_mrm_escalation fires when the report's model is Tier 1 or Tier 2 AND at least one segment carries a RED CALIBRATION finding -- realised outcomes far enough from predicted that the policy calls it degradation, on a segment the policy was willing to judge. It is a BUSINESS condition, not a check on the model: it asks whether this report goes in front of a person this week.

src/extract.py::compute(), called by extract() on every report, by the local app on every click, and by the eval harness on every scored report. It runs on whatever the model returned and NEVER on gold.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is not an escalation. It is a routing signal over a worksheet -- nothing here raises an issue, opens a finding, notifies a committee or tells a supervisor anything.
• It is not any institution's rule. No supervisory guidance, regulation, standard, industry framework or firm's model risk policy was consulted, and none is reproduced.
• It is not a check on the model's reasoning. It reads three fields off the reply and does arithmetic; a reply that is internally inconsistent will produce a flag that is faithfully wrong.
• It is not a severity score. It is one boolean, and a report with six red calibration findings and a report with one look identical to it.

WatchedWhat is watched, and why that one

The kit measures 24 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

5 measured by the latest run 19 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

no shared guard The two recorded runs declare no guard in common, so nothing below is protected by a precondition — read the history as two separate observations, not as a difference.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

3 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked.

not a time series No two of these 3 runs measured the same system — they differ on documents, extraction_cells, failures, provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

DeviationsWhat deviated

0 breaches across 3 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the condition's evidence on any change to compute() in src/extract.py, to the red bands or tier floors in data/policy.json, or to the tier mix in tools/build_corpus.py. The corpus generator reprints the five-candidate firing table on every rebuild, and evals/check_labels.py fails the pre-flight if the shipped condition leaves the 10-45 pct band.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt is empty on purpose: the corpus is generated in-process, the provider is reached over urllib rather than through any vendor SDK, and there was nothing left to add. The hard part of this kit is a four-gate precedence walk over seven numbers -- about ninety lines in src/policy.py -- and every framework that would 'help' with it would move the thresholds out of a file a reader can open.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

The ANSWER is a table; the pipeline is a straight line. One path per report -- segment, select, prompt, one call, parse, span, re-apply the policy in code, compute the flag -- with no branch, no retry loop above the adapter's own backoff and no state carried between reports. The only fan-out is the eval harness running twelve reports at once, and that is a thread pool over a pure function.

The other sideWhat a framework costs you

• The cost of no framework is that the run harness, the record shape and the concurrency are hand-rolled in every kit in this series, and a fix made in one -- the transport-failure retry, the token counters above the parse check -- has to be carried to the others by hand.
• The cost of the policy being data is that a forker can write an incoherent policy and the kit will apply it faithfully. evals/check_labels.py asserts gold re-derives; it does not assert the bands are sensible, and it cannot.
• The cost of one call per report is that a 60-segment pack has no path except a bigger ceiling. There is no chunking seam here and adding one would mean deciding how to split a table whose rows reference each other through the notes.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r002-model-degrade on the fast tier, 2026-09-15. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 2 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

1 run not plotted. b000-model-degrade-tablescan recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the rulebook is data read whole into the prompt.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT -- this repository's own licence. Every institution, model, segment, owner, validator and figure is invented, and each of the 50 files carries a Synthetic Record banner on its first line saying so. No supervisory guidance, regulation, standard, industry framework or firm's model risk policy is reproduced, quoted or named anywhere in the corpus or the policy. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one line⚑ Segment roster -- which segments the report actually monitors

Did the worksheet carry exactly the segments the report monitors, and no others? Micro precision and recall over 294 segments across the 50 reports, matched on the name normalised for case and whitespace only. A segment returned twice counts as a phantom, because a worksheet with two rows for one book is a worksheet somebody has to reconcile before they can read it.

MethodHow the test was run

evals/judge.py::score_roster, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, which records every segment the generator wrote into the report -- including the 26 that appear in the drift table and nowhere else, and the 8 whose two tables use different labels with a note in the prose mapping them.

How tight can the band be? No tolerance band -- a segment is on the worksheet or it is not, compared after collapsing whitespace and case. Case folding is deliberate and it is the ONLY normalisation: it forgives 'WEALTH CASH HUB' for 'Wealth Cash Hub', and it does not forgive 'SME Term' for 'Small Business Term', which is the alias the model is supposed to resolve.

Cadence: Re-score on any change to the segment-name vocabulary in tools/build_corpus.py, to the drift-only or alias counts, or to src/prompt.py's rule 1.

The decisionWhen to reach for it

In one lineReported values, cell by cell

Is every value on the worksheet the value the report states? 2,212 cells on this run: four report-level fields on each of the 49 reports it returned, and seven read values on each of their 288 segments (2,258 across the full 50). THIS IS THE REFERENCE GRADER -- everything else on this page is a judgement about numbers, and this is whether the numbers are right.

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, which holds the value the generator WROTE into the report text -- after the restatement where the notes supersede the table, and normalised to a percentage where the table prints a bare decimal.

How tight can the band be? Tolerance is per field and is set to the precision the report PRINTS: 0.005 on the two rates (printed to 2 dp), 0.0005 on the two AUCs and the PSI (printed to 3 dp), exact on the integer population, exact after case folding on the enums and the identifiers. Looser would forgive a misreading; tighter would convict float noise. The self-test proves both directions: 0.002 on a PSI convicts, 0.0001 does not.

Cadence: Re-score on any change to data/fields.json, to the rate renderings in tools/build_corpus.py, or to the tolerance table at the top of evals/judge.py.

The decisionWhen to reach for it

In one line⚑ The status the run called

For each of the 864 (segment, rule) pairs on the reports the run returned (882 across the full 50), is the status the run RETURNED the status the shipped policy gives? One of within, amber, red, not_reported or not_assessable. The denominator is gold's, so a segment the run never returned loses all three of its findings here.

MethodHow the test was run

evals/judge.py::score_status, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/policy.py::evaluate over gold's own values -- the same function the corpus generator wrote gold with, re-derived and asserted by evals/check_labels.py on all 882 before any run may spend.

How tight can the band be? Exact string match on the status vocabulary. There is no partial credit and no ordinal distance -- amber where gold says red and not_assessable where gold says red are both simply wrong, because they go to different people.

Cadence: Re-score on any change to data/policy.json, to the order of precedence, or to src/prompt.py's rules 8 and 9.

The decisionWhen to reach for it

In one lineThe policy re-applied in code to the run's own numbers

The same 864 comparisons, with the status RE-DERIVED by src/policy.py from the numbers the run itself returned and the tier the run itself read. The gap between this grader and the one above is EXACTLY the run's arithmetic-and-precedence error; the gap between this grader and 100 pct is EXACTLY its reading error. A single accuracy figure hides which of the two you have, and they have completely different fixes -- one is a prompt, the other is eight lines of arithmetic that should never have been asked of a model.

MethodHow the test was run

evals/judge.py::score_status, the code_applied half, in-process.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/policy.py::evaluate over gold's values, exactly as for G3.

How tight can the band be? Same exact-match vocabulary as the grader above.

Cadence: Re-score whenever G3 is re-scored; the two are only meaningful side by side.

The decisionWhen to reach for it

In one lineDegradation found -- the red class alone

Recall and precision on the red class alone -- the 56 findings on the reports this run returned where a segment has degraded past its tier's red band (58 across the full corpus). This is the headline the kit exists to produce, and it is scored separately because a status-accuracy figure dominated by 336 within findings can look excellent while missing every breach.

MethodHow the test was run

evals/judge.py::score_red, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: gold's 58 red findings -- 16 calibration, 16 discrimination, 26 drift. This run was scored against the 56 of them that sit on the 49 reports it returned: 15, 15 and 26.

How tight can the band be? Binary on the class. A red returned as amber is both a miss and, for the amber class, a false positive -- the confusion matrix in the run record carries the full picture.

Cadence: Re-score on any change to the red bands in data/policy.json.

The decisionWhen to reach for it

In one lineThe escalation flag

One boolean per report, computed in pure code from the findings the run returned: is this a Tier 1 or Tier 2 model with at least one red calibration finding? It is a BUSINESS condition and not a check on the model.

MethodHow the test was run

evals/judge.py::score_flag over src/extract.py::compute(), in-process.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: evals/judge.py::gold_flag, the same condition restated over gold's own findings. It fires on 11 of the 50 reports.

How tight can the band be? No band -- the condition is boolean and is computed by src/extract.py::compute() over the reply, never over gold. It returns None rather than False when the reply carries no finding the rule can read; an unknown is not a pass.

Cadence: Re-check the condition's firing band on any change to the red bands, the tier mix, or compute() itself. tools/build_corpus.py prints the five-candidate table on every corpus rebuild.

The decisionWhen to reach for it