The business caseThe problem this solves

A recertification asks a question no single document answers: what CHANGED. Three things on the page look like an answer and are not. An exception recorded this year looks identical whether or not the same control failed last year -- and a second-cycle failure is a different finding entirely, because a whole remediation period has passed. A control missing from this year's report looks identical whether the vendor stopped operating it or the report's SCOPE shrank to exclude its whole domain. And when the service auditor changes, the incoming firm renumbers AND rewords every control, so a reader who lines the two reports up by control id reports one control as one removal plus one addition -- and the worksheet looks complete. Someone opening two assurance reports side by side at recertification time and working out what moved: which exceptions are new, which have now run for a second cycle, which controls the current report no longer covers, and whether the opinion slipped. It is an hour or two per third party and the part that goes wrong is not the reading -- it is reading only this year's report, because that is the one that arrived.

The inputThe actual recertification packets

The corpus is 50 recertification packets, 0.40 MB (txt 50). ⚑ THE UNIT IS A PAIR, AND FOUR SHAPES ARE DEALT IN COUNTED QUANTITIES rather than left to chance. 100 rows across 10 packets are controls the two reports NUMBER DIFFERENTLY, because the service auditor changed -- a reader who joins on the id returns each of them TWICE. 8 packets exclude a whole domain from the current report's scope, so 19 controls are absent this year for a reason that says nothing about the vendor. 8 rows reuse one control id for a materially different control. And 4 packets have NO PRIOR REPORT at all -- first assurance cycle -- which is where the selection guard earns its place. evals/check_labels.py asserts a floor for every one of those shapes and refuses to let a run spend if the corpus has drifted below it, and it asserts two things about the DIFFICULTY as well: that the pure-code id split produces 705 rows against gold's 605, and that an exact-description join rescues 0 of the 100 renumbered rows.

Synthetic Record
----------------
This recertification packet is SYNTHETIC and was generated for a public demonstration kit.
The vendor, the service, the audit firms, the people, the sites, the contract, the fee and
every control and exception in it are invented. No real third party, service auditor,
assurance report or recertification file is reproduced or referred to. The criteria set the
reports cite (ARC-4) does not exist. Nothing here is an assurance opinion, a recertification
decision or third-party risk advice.


Recertification Request
-----------------------
PKT-0001
Vendor reference: VEN-4100
Vendor: Marlowe Clearing Services
Service in scope: card authorisation switching
Recertification due: 3 February 2027
Prepared for: Third-Party Risk Management
Prior assurance report held: yes

Vendor Contact File
-------------------
Relationship manager: A. Redgrave
Direct line: +44 20 7879 3259
Email: redgrave@marlowe.example
Master services agreement: MSA-48415
Annual contracted fee: GBP 99,000
Internal cost centre: CC-5975

Current Report Header
---------------------
Report period: 1 October 2025 to 30 September 2026
Service auditor: Hallam Rowe LLP
Report type: Type II
Criteria set: ARC-4 (illustrative - see the banner above)
Opinion on operating effectiveness: qualified
Domains excluded from the scope of this report: none
Subservice organisations carved out: none

Current Control ARC-1.1
-----------------------
Domain: access
Description: Access for leavers is withdrawn from card authorisation switching within one working day of the leaving date.
Test result: clean - no exceptions noted
Exception detail: none recorded.

Current Control ARC-2.2
-----------------------
Domain: change_management

The outcomeWhat a good result looks like

One worksheet row per control in the union of the two reports, each carrying the change call the shipped rulebook makes from three stated facts, plus a packet-level opinion call and a pure-code flag for the one finding a reader of this year's report cannot reach.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Reading this year's assurance report against last year's at recertification, as a first pass for a third-party risk reviewer to work through — the one tier measured here — 605 of 605 controls joined with 0 phantoms, 100 of 100 renumberings resolved, and every change call right
  Against the free id join on the same 50 packets: 0 phantom rows against 100, 100 of 100 renumberings resolved against 0, 0 over-confident rows against 8, and a review flag with 0 false alarms against 7.

And where nothing here is good enough:

• Deciding a recertification, a renewal, a suspension or a termination — nothing here
  The kit produces a comparison worksheet and no code path in it decides anything: src/compare.py returns a JSON body, src/app.py serves one, and neither writes, approves, notifies or escalates to anybody.
• Comparing three or more years, or reports whose control tables are in PDF tables rather than headed text sections — not this kit as it stands
  A two-year comparison is the whole design — the prompt, the rulebook's three stated facts and the aligner's three-way split all assume exactly two reports. And src/segment.py cuts on underlined Current Control <id> / Prior Control <id> headings, which is a property of this generated corpus's layout.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-22. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-22 — r001-vendor-recert. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — git clone, python -m tools.build_corpus (0.03s), python -m evals.check_labels (0.09s), python -m evals.run --run-id t000-vendor-recert-stub --stub (0.14s) and python -m evals.run --run-id b000-vendor-recert-idjoin --baseline (0.12s) -- all four free, no key, no network, no install. Measured on 2026-08-22.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per recertification packet

Current processWhat it replaces

Someone opening two assurance reports side by side at recertification time and working out what moved: which exceptions are new, which have now run for a second cycle, which controls the current report no longer covers, and whether the opinion slipped. It is an hour or two per third party and the part that goes wrong is not the reading -- it is reading only this year's report, because that is the one that arrived.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptArchitectureArchitectureApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 1 of 8 stations carry a recorded failure

scroll → 8 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one packet section is

cut on underlined headings -- one section per control per YEAR, plus the recertification request, the two report headers, the vendor contact file and the closing note; a packet with no headings falls back to one whole-document segment. There is no train/test split because nothing is trained: all 50 packets are scored.

SetupWhat the setup figure measured

There is no index and no retrieval. Preparation is segmentation plus a pure-code id split -- 50 packets cut into 1366 sections, in process, with no model and no network, in the 0.03 seconds tools/build_corpus.py takes to write the whole corpus. 1316 of those sections are sent; the 50 that are not are exactly the Vendor Contact File sections, which carry a named relationship manager, a direct line, an email address, the master services agreement number and the ANNUAL CONTRACTED FEE, and which no field asks for.

LicenceLicence

MIT -- this repository's own licence. Every vendor, service, audit firm, person, site, telephone number, email address, contract, fee, control, control id, exception and opinion is invented; no real third party, service auditor, assurance report or recertification file is reproduced or referred to, and no assurance standard, attestation framework, regulator, supervisory body or industry group is named. The rulebook shipped with this kit is ILLUSTRATIVE and is not an authority. It was written for this kit and reproduces no assurance standard, attestation framework, regulator's third-party risk guidance, supervisory expectation, industry body guidance or firm's own vendor recertification procedure, and it names none of them. The criteria set the corpus cites (ARC-4) does not exist.

Bring your ownBring your own recertification packets

Drop your own packets into data/corpus/ as .txt with the same section headings, delete data/gold.jsonl, and the free paths still work: python -m src.app renders and python -m evals.run --baseline scores nothing but produces the id-join worksheet. To SCORE your own you need gold, and gold here is not typed -- it is the rulebook's lookup over three facts, so labelling is: read prior_state, current_state and comparability off each control and let src/rulebook.py compute the rest. If your headings differ, change the six constants at the top of src/select.py and the two prefixes in src/align.py; nothing else in the kit knows what a section is called.

⚠︎ And what stops being true when you do: ⚠︎ Vendor Contact File NEVER REACHES A PROVIDER, and on your own packets that guard protects whatever you put in it. The section is mapped by no field, and _fallback() in src/select.py subtracts NEVER_SENT even when a hint matches nothing -- which is not hypothetical here: prior_opinion and prior_state map ONLY to the prior report, and 4 packets have no prior report, so the plain whole-document fallback every sibling kit shipped sends the contact file on those 4. Measured over all 50 packets: 0 leaks with the guard, 4 without it.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 4,805 tokens assembled, 1,818 are documents — 38% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured token-for-token via evals/prompt_tokens.py's nested-prefix method on PKT-0014 -- the same packet the worked example above was captured on, so the five parts sum to exactly the 4805 input tokens that call actually reported. Five calls at max_tokens=1, each part's size the difference between two consecutive prompt_tokens counts the provider itself returned. results/tokens-p001-vendor-recert.json.

Full promptThe prompt in full

You read a RECERTIFICATION PACKET -- one vendor's CURRENT assurance report and, usually, the PRIOR YEAR'S report for the same vendor -- and return a REVIEWER'S COMPARISON WORKSHEET: what MOVED between the two years. You return JSON and nothing else.

You are preparing a worksheet for a qualified reviewer. You never recertify a vendor, never approve, suspend, terminate or renew anything, never raise a finding and never clear one. Your job is to say what the two reports state, apply the rulebook given below, and NAME THE CONTROLS THE TWO REPORTS DO NOT LET YOU COMPARE.

RULES, in order of importance:
1. ONE ROW PER CONTROL IN THE UNION OF THE TWO REPORTS, AND NO OTHERS. A control tested in either year gets exactly ONE row. Never return the same control twice.
2. THE CANDIDATE ID SPLIT BELOW IS PURE CODE AND CAN BE WRONG. If the service auditor changed, the new firm renumbered everything: a control tested in BOTH years appears once in the current-only list and once in the prior-only list, and it is ONE row. Read the DESCRIPTIONS. When they describe the same control, return one row whose `control_ref` is the CURRENT id and whose `prior_ref` is the PRIOR id.
3. `control_ref` IS THE CURRENT REPORT'S ID where the control appears this year, and the PRIOR report's id where it does not. `prior_ref` is the prior id ONLY when it differs from `control_ref`; otherwise it is the literal string `none`.
4. AN EXCEPTION RECORDED IN BOTH YEARS IS `repeat_exception`, NEVER `new_exception`. A whole remediation cycle has passed and the control is still failing. This is the call a reader of one report cannot make, and getting it wrong is the most expensive mistake on this worksheet.
5. A CONTROL ABSENT THIS YEAR BECAUSE THE SCOPE SHRANK IS NOT A DROPPED CONTROL. The current report's header lists the domains it excludes. A prior control in an excluded domain is `comparability` 'scope_withdrawn', which the rulebook turns into `not_comparable`. Its absence says nothing about whether the vendor still operates it.
6. THE SAME ID CAN NAME TWO DIFFERENT CONTROLS. Compare the two DESCRIPTIONS before you compare the two results. Where they are materially different controls and no other description in the other report matches, `comparability` is 'id_reused' and the row is `not_comparable`.
7. A NOTE IN THE PRIOR REPORT THAT SOMETHING WAS 'remediated after the period end' SETTLES NOTHING. Only what the CURRENT report records decides this year's state.
8. `change_kind` IS DECIDED ONLY BY THE RULEBOOK BELOW, from `prior_state`, `current_state` and `comparability`. Work through the eight steps IN ORDER and STOP at the first that fires.
9. `opinion_movement` IS DECIDED ONLY BY THE RULEBOOK, from `prior_report_present`, `prior_opinion` and `current_opinion`. It is asked once per packet and is independent of every control row.
10. `not_comparable` IS A REAL ANSWER AND YOU ARE EXPECTED TO USE IT. A worksheet that never reaches for it is guessing, and a change recorded confidently on two reports that do not join is a finding nobody can check.
11. Copy every control id verbatim from the packet. Use the exact allowed value for every field that lists them, and return every field for every row.

RECERTIFICATION COMPARISON RULEBOOK (the authority for `change_kind` and `opinion_movement`; this is an ILLUSTRATIVE rulebook written for this kit, and it reproduces no assurance standard, attestation framework, regulator's third-party risk guidance or firm recertification procedure)

WHAT IS ON THE WORKSHEET
  - ONE ROW PER CONTROL IN THE UNION OF THE TWO REPORTS. A control tested in either year gets exactly one row, never two.
  - `control_ref` is the CURRENT report's control id when the control appears in the current report, and the PRIOR report's id when it does not. That is the row's key and it is unique within a packet.
  - `prior_ref` is the PRIOR report's id for the same control, and only when that id DIFFERS from `control_ref`. Otherwise it is the literal string `none`.
  - A control the two reports number differently is ONE row, not an addition and a removal. Resolving that is the job.
  - Nothing here recertifies, approves, suspends, terminates or renews anything. It is a reviewer's comparison worksheet.

CONTROL DOMAINS
  access             who can reach the service and how that is granted, reviewed and withdrawn
  change_management  how changes to the service are approved, reviewed and released
  operations         how the service is run day to day - processing, incidents, reconciliation, capacity
  subservice         what the vendor's own suppliers do, and what the vendor tells its customers to do
  physical           physical access to the site the service runs from
  resilience         backup, restore and continuity

THE STATED FACTS, read off the two reports for every control
  prior_state:
    clean              the prior report tested this control and recorded no exception
    exception          the prior report tested this control and recorded at least one exception
    not_tested         the prior report lists this control and records that it was not tested in the period
    absent             the prior report does not carry this control at all
  current_state:
    clean              the current report tested this control and recorded no exception
    exception          the current report tested this control and recorded at least one exception
    not_tested         the current report lists this control and records that it was not tested in the period
    absent             the current report does not carry this control at all
  comparability:
    comparable         the two reports can be read against each other for this control
    scope_withdrawn    the control sits in a domain the CURRENT report's header excludes from its scope, so its absence this year says nothing about the vendor
    no_prior_report    no prior report is held for this vendor, so there is nothing to read the current one against
    id_reused          the same control id names a materially DIFFERENT control in the two reports, and no description in the other report matches it, so the two ends do not join

THE CHANGE CALL -- work through IN ORDER, stop at the first that fires
  1. If `comparability` is anything other than `comparable`, the answer is `not_comparable`. Stop. Nothing below is asked.
  2. If `prior_state` is `absent`, the answer is `control_added`.
  3. If `current_state` is `absent`, the answer is `control_dropped`.
  4. If `current_state` is `exception` AND `prior_state` is `exception`, the answer is `repeat_exception`.
  5. If `current_state` is `exception`, the answer is `new_exception`.
  6. If `current_state` is `not_tested` AND `prior_state` is not `not_tested`, the answer is `testing_withdrawn`.
  7. If `prior_state` is `exception` AND `current_state` is `clean`, the answer is `exception_cleared`.
  8. Otherwise the answer is `no_change`.

WHAT EACH CHANGE KIND MEANS
  no_change          the control was tested both years and the result did not move
  new_exception      an exception this year where the prior report recorded none
  repeat_exception   an exception this year AND last year - the one a single-year read cannot tell from a new one
  exception_cleared  an exception last year, clean this year
  control_dropped    the prior report tested it, the current report does not carry it, and the scope did not change to explain that
  control_added      the current report carries it and the prior report did not
  testing_withdrawn  the control is still listed this year and the report records that it was not tested
  not_comparable     the two ends do not join - see `comparability` for which of the three reasons

THE OPINION CALL -- one per packet, independent of every control call
  opinion rank, worst last: unqualified=0, qualified=1, adverse=2, disclaimer=3
  1. If `prior_report_present` is `no`, the answer is `no_prior_report`. Stop.
  2. If either opinion is `none_stated` or `absent`, the answer is `not_comparable`. Stop.
  3. If the two opinions rank equal, the answer is `unchanged`.
  4. If the current opinion ranks HIGHER than the prior one, the answer is `worsened`.
  5. Otherwise the answer is `improved`.
  A `disclaimer` ranks worst of the four: an auditor who could not form an opinion has told you less than one who formed a bad one.

WHY `not_comparable` IS A REAL ANSWER
  A recertification worksheet exists to say what MOVED. `not_comparable` says the paperwork does not let you tell, which is a finding about the reports and not a gap in the answer. A row filled in confidently where the two reports do not join is the expensive mistake here: it puts a change on a reviewer's list that nothing in the file supports, or - worse - it records `no_change` for a control nobody looked at.

CANDIDATE ALIGNMENT BY CONTROL ID (pure code, and it can be WRONG -- see below)
  ids in BOTH reports:       (none)
  ids in the CURRENT report only: C-009, C-012, C-013, C-015, C-017, C-019, C-020, C-023, C-024, C-026
  ids in the PRIOR report only:   AC-01, CM-01, OP-01, OP-02, PH-01, PH-02, RS-01, RS-02, SS-01, SS-02
  This split was made by matching id strings and NOTHING ELSE. It is a starting point, not an answer.
  * If the service auditor changed, the new firm renumbered everything: a control tested in both years appears once in each of the two 'only' lists, and the DESCRIPTIONS are what tell you they are one control.
  * An id in BOTH lists can still name two DIFFERENT controls. Read the two descriptions before you compare the two results.

Return these:
- packet_id (string) -- the packet reference printed in the Recertification Request section, verbatim
- vendor_ref (string) -- the vendor reference printed in the Recertification Request section, verbatim
- prior_report_present (string) one of: yes, no -- does this packet carry a PRIOR report at all
- auditor_changed (string) one of: yes, no, not_determined -- do the two report headers name DIFFERENT service auditors; not_determined when there is no prior report
- scope_changed (string) one of: yes, no, not_determined -- does the CURRENT report's header exclude a domain the prior report did not exclude; not_determined when there is no prior report
- current_opinion (string) one of: unqualified, qualified, adverse, disclaimer, none_stated, absent -- the opinion on operating effectiveness in the CURRENT report header
- prior_opinion (string) one of: unqualified, qualified, adverse, disclaimer, none_stated, absent -- the opinion on operating effectiveness in the PRIOR report header; `absent` when there is no prior report
- opinion_movement (string) one of: unchanged, worsened, improved, not_comparable, no_prior_report -- THE OPINION CALL. Decided ONLY by the rulebook, from the two opinions above
- controls (array of objects) -- one object per control in the UNION of the two reports, each carrying:
    - control_ref (string) -- the CURRENT report's control id where the control appears this year, otherwise the PRIOR report's id. Copy it verbatim. It is the row key and must be unique in the packet
    - prior_ref (string) -- the PRIOR report's id for the SAME control, and only when it differs from control_ref. Otherwise the literal string `none`
    - domain (string) one of: access, change_management, operations, subservice, physical, resilience -- the domain the CURRENT report gives this control, or the PRIOR report's where the control is not in the current report
    - prior_state (string) one of: clean, exception, not_tested, absent -- what the PRIOR report records for this control
    - current_state (string) one of: clean, exception, not_tested, absent -- what the CURRENT report records for this control
    - comparability (string) one of: comparable, scope_withdrawn, no_prior_report, id_reused -- whether the two reports can be read against each other for THIS control
    - change_kind (string) one of: no_change, new_exception, repeat_exception, exception_cleared, control_dropped, control_added, testing_withdrawn, not_comparable -- THE CHANGE CALL. Decided ONLY by the rulebook, from prior_state, current_state and comparability

Return a JSON object with exactly these top-level keys: packet_id, vendor_ref, prior_report_present, auditor_changed, scope_changed, current_opinion, prior_opinion, opinion_movement, controls
`controls` is an array. Return it empty only if neither report in this packet carries a single control.

RECERTIFICATION PACKET
----------------------
Synthetic Record
----------------
This recertification packet is SYNTHETIC and was generated for a public demonstration kit.
The vendor, the service, the audit firms, the people, the sites, the contract, the fee and
every control and exception in it are invented. No real third party, service auditor,
assurance report or recertification file is reproduced or referred to. The criteria set the
reports cite (ARC-4) does not exist. Nothing here is an assurance opinion, a recertification
decision or third-party risk advice.

Recertification Request
-----------------------
PKT-0014
Vendor reference: VEN-4194
Vendor: Slape Hill Contact Centre
Service in scope: outsourced inbound servicing
Recertification due: 9 December 2026
Prepared for: Third-Party Risk Management
Prior assurance report held: yes

Current Report Header
---------------------
Report period: 1 October 2025 to 30 September 2026
Service auditor: Vantry & Co
Report type: Type II
Criteria set: ARC-4 (illustrative - see the banner above)
Opinion on operating effectiveness: unqualified
Domains excluded from the scope of this report: none
Subservice organisations carved out: none

Current Control C-009
---------------------
Domain: access
Description: A quarterly recertification of user access to outsourced inbound servicing is performed and removals are evidenced.
Test result: EXCEPTION NOTED
Exception detail: 2 selected instances were completed after the stated deadline, the longest by 11 working days.

Current Control C-012
---------------------
Domain: resilience
Description: Backup of outsourced inbound servicing runs to a defined schedule and restoration is tested with evidence retained.
Test result: EXCEPTION NOTED
Exception detail: The control operated for 7 of the 12 months in the period; no evidence was retained for the remainder.

Current Control C-013
---------------------
Domain: change_management
Description: Every code release to outsourced inbound servicing carries evidence of a second engineer's review.
Test result: EXCEPTION NOTED
Exception detail: 3 of 38 items sampled for the period were not evidenced.

Current Control C-015
---------------------
Domain: access
Description: Access to outsourced inbound servicing is provisioned only where the service owner has recorded an approval.
Test result: not tested in the period
Exception detail: not applicable - the service auditor records that this control was not tested in the period.

Current Control C-017
---------------------
Domain: subservice
Description: Third-party assurance reports covering the suppliers behind outsourced inbound servicing are collected and reviewed each year.
Test result: EXCEPTION NOTED
Exception detail: 2 selected instances were completed after the stated deadline, the longest by 13 working days.

Current Control C-019
---------------------
Domain: operations
Description: Utilisation of outsourced inbound servicing is tracked against defined thresholds and reported each month.
Test result: not tested in the period
Exception detail: not applicable - the service auditor records that this control was not tested in the period.

Current Control C-020
---------------------
Domain: operations
Description: Scheduled processing for outsourced inbound servicing is monitored, and any failed job is restarted under a documented instruction.
Test result: clean - no exceptions noted
Exception detail: none recorded.

Current Control C-023
---------------------
Domain: change_management
Description: The author of a change to outsourced inbound servicing may not be the person who deploys it.
Test result: EXCEPTION NOTED
Exception detail: The control operated for 7 of the 12 months in the period; no evidence was retained for the remainder.

Current Control C-024
---------------------
Domain: subservice
Description: Customers of outsourced inbound servicing are notified in writing of the controls they are expected to operate themselves.
Test result: clean - no exceptions noted
Exception detail: none recorded.

Current Control C-026
---------------------
Domain: subservice
Description: Supplier performance against the agreed outsourced inbound servicing service levels is examined quarterly.
Test result: EXCEPTION NOTED
Exception detail: Evidence was produced for 8 of 9 selections; the remainder could not be located.

Prior Report Header
-------------------
Report period: 1 October 2024 to 30 September 2025
Service auditor: Sarn Ferris Assurance
Report type: Type II
Criteria set: ARC-4 (illustrative - see the banner above)
Opinion on operating effectiveness: qualified
Domains excluded from the scope of this report: none
Subservice organisations carved out: none

Prior Control AC-01
-------------------
Domain: access
Description: Logical access to outsourced inbound servicing is granted only on documented approval from the service owner.
Test result: EXCEPTION NOTED
Exception detail: 1 of 32 items sampled for the period were not evidenced. The service organisation states that the matter was remediated after the period end.

Prior Control CM-01
-------------------
Domain: change_management
Description: Code released to outsourced inbound servicing is peer reviewed and the review is evidenced.
Test result: EXCEPTION NOTED
Exception detail: The supporting register was not maintained for 1 of the 4 quarters in the period.

Prior Control OP-01
-------------------
Domain: operations
Description: Incidents affecting outsourced inbound servicing are logged, categorised and closed against a stated resolution.
Test result: EXCEPTION NOTED
Exception detail: 2 selected instances were completed after the stated deadline, the longest by 9 working days. The service organisation states that the matter was remediated after the period end.

Prior Control OP-02
-------------------
Domain: operations
Description: Capacity for outsourced inbound servicing is monitored against stated thresholds and reported monthly.
Test result: not tested in the period
Exception detail: not applicable - the service auditor records that this control was not tested in the period.

Prior Control PH-01
-------------------
Domain: physical
Description: Visitors to the Denholm processing floor are escorted throughout and the escort is recorded.
Test result: EXCEPTION NOTED
Exception detail: The control operated for 7 of the 12 months in the period; no evidence was retained for the remainder. The service organisation states that the matter was remediated after the period end.

Prior Control PH-02
-------------------
Domain: physical
Description: Physical access to the Denholm processing floor is granted on documented approval and logged on entry.
Test result: EXCEPTION NOTED
Exception detail: Evidence was produced for 18 of 19 selections; the remainder could not be located. The service organisation states that the matter was remediated after the period end.

Prior Control RS-01
-------------------
Domain: resilience
Description: Backups of outsourced inbound servicing are taken to a stated schedule and restore tests are evidenced.
Test result: EXCEPTION NOTED
Exception detail: 4 of 39 items sampled for the period were not evidenced. The service organisation states that the matter was remediated after the period end.

Prior Control RS-02
-------------------
Domain: resilience
Description: Recovery time objectives for outsourced inbound servicing are stated and measured against actual recovery.
Test result: clean - no exceptions noted
Exception detail: none recorded.

Prior Control SS-01
-------------------
Domain: subservice
Description: Assurance reports from subservice organisations supporting outsourced inbound servicing are obtained and reviewed annually.
Test result: EXCEPTION NOTED
Exception detail: The supporting register was not maintained for 1 of the 4 quarters in the period.

Prior Control SS-02
-------------------
Domain: subservice
Description: Subservice organisation performance against the outsourced inbound servicing service levels is reviewed each quarter.
Test result: clean - no exceptions noted
Exception detail: none recorded.

Packet Notes
------------
Assembled by the third-party risk team for the recertification file. The reviewer is asked to read the two reports against each other and record what moved: the opinion, any exception that is new, any exception that has now run for a second cycle, and any control the current report no longer covers. The service auditor changed between the two periods and the new firm uses its own control numbering, so the two reports do not line up by control id. Nothing in this packet is a recertification decision; the recertification is the risk team's, taken on the whole file.

Raw responseThe raw response

{
  "packet_id": "PKT-0014",
  "vendor_ref": "VEN-4194",
  "prior_report_present": "yes",
  "auditor_changed": "yes",
  "scope_changed": "no",
  "current_opinion": "unqualified",
  "prior_opinion": "qualified",
  "opinion_movement": "improved",
  "controls": [
    {
      "control_ref": "C-009",
      "prior_ref": "none",
      "domain": "access",
      "prior_state": "absent",
      "current_state": "exception",
      "comparability": "comparable",
      "change_kind": "control_added"
    },
    {
      "control_ref": "C-012",
      "prior_ref": "RS-01",
      "domain": "resilience",
      "prior_state": "exception",
      "current_state": "exception",
      "comparability": "comparable",
      "change_kind": "repeat_exception"
    },
    {
      "control_ref": "C-013",
      "prior_ref": "CM-01",
      "domain": "change_management",
      "prior_state": "exception",
      "current_state": "exception",
      "comparability": "comparable",
      "change_kind": "repeat_exception"
    },
    {
      "control_ref": "C-015",
      "prior_ref": "AC-01",
      "domain": "access",
      "prior_state": "exception",
      "current_state": "not_tested",
      "comparability": "comparable",
      "change_kind": "testing_withdrawn"
    },
    {
      "control_ref": "C-017",
      "prior_ref": "SS-01",
      "domain": "subservice",
      "prior_state": "exception",
      "current_state": "exception",
      "comparability": "comparable",
      "change_kind": "repeat_exception"
    },
    {
      "control_ref": "C-019",
      "prior_ref": "OP-02",
      "domain": "operations",
      "prior_state": "not_tested",
      "current_state": "not_tested",
      "comparability": "comparable",
      "change_kind": "no_change"
    },
    {
      "control_ref": "C-020",
      "prior_ref": "none",
      "domain": "operations",
      "prior_state": "absent",
      "current_state": "clean",
      "comparability": "comparable",
      "change_kind": "control_added"
    },
    {
      "control_ref": "C-023",
      "prior_ref": "none",
      "domain": "change_management",
      "prior_state": "absent",
      "current_state": "exception",
      "comparability": "comparable",
      "change_kind": "control_added"
    },
    {
      "control_ref": "C-024",
      "prior_ref": "none",
      "domain": "subservice",
      "prior_state": "absent",
      "current_state": "clean",
      "comparability": "comparable",
      "change_kind": "control_added"
    },
    {
      "control_ref": "C-026",
      "prior_ref": "SS-02",
      "domain": "subservice",
      "prior_state": "clean",
      "current_state": "exception",
      "comparability": "comparable",
      "change_kind": "new_exception"
    },
    {
      "control_ref": "OP-01",
      "prior_ref": "none",
      "domain": "operations",
      "prior_state": "exception",
      "current_state": "absent",
      "comparability": "comparable",
      "change_kind": "control_dropped"
    },
    {
      "control_ref": "PH-01",
      "prior_ref": "none",
      "domain": "physical",
      "prior_state": "exception",
      "current_state": "absent",
      "comparability": "comparable",
      "change_kind": "control_dropped"
    },
    {
      "control_ref": "PH-02",
      "prior_ref": "none",
      "domain": "physical",
      "prior_state": "exception",
      "current_state": "absent",
      "comparability": "comparable",
      "change_kind": "control_dropped"
    },
    {
      "control_ref": "RS-02",
      "prior_ref": "none",
      "domain": "resilience",
      "prior_state": "clean",
      "current_state": "absent",
      "comparability": "comparable",
      "change_kind": "control_dropped"
    }
  ]
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch what changed in a vendor's report, year to year — 50 recertification packets. One model answered, and every answer was then graded Seven different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

Two runs of the same 50 packets are published side by side and they are NOT two models: r001 is the shipped prompt and a001 is the same prompt with its five trap-naming rules and the pure-code alignment block removed. Both were fired once, at the same ceiling, against the same corpus and the same judge. Nothing here ranks two models, because only one tier was run.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18.

The gradersSeven ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The free id-join floor (evals/baseline.py) and the scorer (evals/judge.py) are both pure code and cost $0.00 to run against any result set. The figure above is the ONE scored run's own token counts (50 packets) priced at Google Gemini 3 Flash's published rate -- the same basis cost_per_query_usd uses. ⚠︎ IT IS NOT WHAT BUILDING THIS KIT COST. Adding the 6-packet calibration, the 50-packet ablation and the worked-example capture, this kit spent 470,159 input and 353,561 output tokens across 107 billed calls, which projects to $1.2958 on the same card -- roughly 2.1 times the scored run alone, and the ablation is most of the difference. ⚠︎ AND THE CALL LEDGER RECORDS 114, NOT 107. The other 7 are two UI screenshot clicks and five prompt-split calls fired at max_tokens=1 to measure the token breakdown; they are billed, they are tiny, and their tokens are NOT in the figure above because no result file records them. The ledger is the authority on how many calls were made; the result files are the authority on what they cost.

Cost driversWhat actually moves the bill

• ⚑ PROVIDER-SIDE REASONING, WHICH IS MOST OF THE BILL. 124,285 of the run's 165,995 output tokens (74.9 pct) were reasoning rather than the JSON worksheet. Output is 81 pct of the cost on the projected card. Reasoning was left at the provider's default and nothing here has measured what turning it off would do to the calls.
• THE FIXED PREFIX, WHICH IS MOST OF THE INPUT. The system prompt, the whole comparison rulebook and the field schema are byte-identical on every call and come to 2754 of the worked example's 4805 input tokens — 57 pct — before either report is read. The rulebook alone is 1292 tokens, and it is the part that grows when a forker replaces it with their own procedure.
• THE SIZE OF THE UNION. Both reports go into one call and the reply is one row per control in the union, so a vendor with sixty controls is roughly five times this input and a longer reply. These packets carry 9 to 14.
• WHETHER THE AUDITOR CHANGED. The renumbered packets are where the reasoning goes: the calibration's slowest call was 72425 ms on an auditor-change packet against a 25029 ms median, because every unpaired control has to be read against every other one.

Your volumeWhat it costs at your volume

Linear in packets: each call is independent, carries the same fixed prefix and shares nothing with its neighbours, so 500 packets cost ten times 50. Nothing amortises -- there is no index and no cache. ⚠︎ AND IT IS NOT LINEAR IN THE SIZE OF THE REPORT, which is the axis that actually moves. Both years go into one call, so a report with sixty controls instead of twelve is roughly five times the input AND a longer reply AND a larger cross-product for the model to resolve; 2987 of every 4805 input tokens on this corpus are the fixed prefix, and that share falls away as the reports grow.

Model choiceWhy this model

One tier was run, so this kit ranks nothing. The tier was chosen because it is what the shared connection in this repository points at, and every sibling kit in this series measures on the same one so their numbers are comparable with each other. ⚠︎ AND THIS KIT CANNOT TELL YOU WHETHER A CHEAPER ONE WOULD DO, which is the question a reader actually has: it scored 100 pct on every grader, so there is no headroom left in this corpus to rank a second model against the first. A cheaper tier could only tie.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 50 recertification packets, one completion call each, one tier. The calibration, the ablation, the worked-example capture, the prompt-split calls and the UI screenshot are recorded separately in Cost.cost_of_evaluation_usd and are not projected here.

The codeEvery file, and what it is for

Eight modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Six of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pysegment

cut the packet into addressable sections, pure code -- one per control per year, plus the request, the two report headers, the contact file and the closing note. 1366 sections across the 50 packets

# Cut a trace file into addressable sections. Pure code — no model, no network.

def sections(text):
def locate(text, value):
def span_label(secs, start):

src/align.pyalign — a swap seam

split the control ids three ways -- in both reports, current only, prior only -- and hand it to the model LABELLED AS A CANDIDATE THAT CAN BE WRONG. Pure code, and deliberately no description matching: on the 10 auditor-change packets this split is wrong on every row, and resolving it is the measured question

You change it to: how much of the join is done in pure code before the model sees it. Today it is an exact id split and nothing else, on purpose; a fuzzy description matcher goes here and changes what the measurement means

# THE ALIGN STATION -- candidate pairings between the two years. Pure code, no model, no network.

def _ref(name, prefix):
def split(secs):
def block(secs):

src/select.pyselect — a swap seam

choose which sections reach the model. Both reports in full; Vendor Contact File never, on any of the 50 packets

You change it to: which sections may never reach a provider. NEVER_SENT is subtracted unconditionally by _fallback, including when a hint matches nothing -- add your own section names to that tuple and no field can pull them onto the wire

# Pick which sections of a recertification packet are sent. Pure code -- the last deterministic

BANNER = "Synthetic Record"
REQUEST = "Recertification Request"
CURRENT_HEADER = "Current Report Header"
PRIOR_HEADER = "Prior Report Header"
NOTES = "Packet Notes"
CURRENT_CONTROL = "Current Control "
PRIOR_CONTROL = "Prior Control "
CONTROL_PREFIXES = (CURRENT_CONTROL, PRIOR_CONTROL)
NEVER_SENT = ("Vendor Contact File",)

src/prompt.pyprompt

assemble one prompt per packet -- system, the whole comparison rulebook, the candidate alignment, the field schema and the packet's sections. 4805 input tokens on the worked example

# Assemble the recertification prompt. ONE prompt per packet, the whole comparison in the reply.

def _system(bare=False):
SYSTEM = _system()
RULEBOOK_TEXT = rulebook_block()
SYSTEM_BARE = _system(bare=True)
def field_schema(packet_fields, control_fields):
def build(doc_text, secs, fields, selector, ablate=None):
def parse(raw, fields):

src/adapters/__init__.pyone model call — a swap seam

one completion per packet, one provider, one key, at max_tokens=32000

You change it to: provider and model. PROVIDER picks the adapter, BASE_URL picks the endpoint, MODEL picks the model; adding a vendor is one function and one entry in PROVIDERS

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

src/compare.pyconfidence floor

hold() -- re-run the rulebook over each row's OWN stated facts and HOLD the row where the two disagree. Pure code. It abstains; it never corrects, because a published accuracy for a corrected answer would be the rulebook's score and not the reading's

# Compare one recertification packet's two reports: segment, align, select, prompt, one model

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 32000
def load_fields():
def control_fields(fields=None):
def packet_fields(fields=None):
def load_doc(doc_id):
def documents():

src/compare.pyreview flag

compute() -- a pure-code business condition over whatever the model returned: a control the prior report tested that the current one does not carry, with no scope change to explain it

# Compare one recertification packet's two reports: segment, align, select, prompt, one model

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 32000
def load_fields():
def control_fields(fields=None):
def packet_fields(fields=None):
def load_doc(doc_id):
def documents():

evals/judge.pythe scorer

six graders, never folded together, over a gold that is the rulebook's own lookup. No model call, so re-scoring costs nothing

# Score a recertification-comparison run. PURE CODE -- gold is exact and every answer is one value

PACKET_CELLS = ("packet_id", "vendor_ref", "prior_report_present", "auditor_changed",
CELL_FIELDS = ("control_ref", "prior_ref", "domain", "prior_state", "current_state",
SPANNABLE = ("control_ref", "prior_ref")
def norm(v):
def code_norm(v):
def equal(got, want):
def _rows_of(rec):
def score_join(records, golds):
def score(fields, records, golds):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 4641 input and 3319 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's recertification packets are entirely synthetic (tools/build_corpus.py, seed 20260822): no real third party, service auditor, assurance report, person, site, contract, fee, telephone number or email address exists in the corpus, and no assurance standard, attestation framework, regulator or supervisory body is named. Every file carries a Synthetic Record banner on its first line and evals/check_labels.py fails if one is missing. The only personal-looking data in the corpus — a relationship manager's name, a direct line and an email address — sits in the one section that never leaves the machine.

Read from the shared .env, this kit's own .env or the real environment only, never committed, never logged and never rendered. Both .env files are gitignored from the first commit. src/app.py redacts the key and the base URL out of any provider error before returning it to the page.

The experimentWe did not attack it — and one boundary was broken until the gate caught it

Four boundaries hold. Two were confirmed by reading the code: no code path recertifies, notifies or writes anything, and no reply that was cut off can be recorded as free because the token counters sit above the continue. The third is enforced by a shared budget checked in the adapter rather than the harness. The fourth — the one that matters — is the never-sent section, and it was NOT holding when it was first written: the whole-document fallback every sibling kit ships sends the contact file on the four first-cycle packets. evals/check_labels.py now audits all 50 packets in both directions on every invocation, and refuses to pass if the guard is not demonstrably the reason nothing leaks. Confirmed by reading the code and by a pre-flight gate that convicted it once, not by an attack run, on 2026-08-22 — no attack run was fired; see redteam.why.

Four boundaries hold and the second is the interesting one, because it did not hold when it was written. The fallback path every sibling kit shipped sends the contact file — with the annual contracted fee in it — on the four first-cycle packets, and every published figure is identical either way. No grader on this page could see it; only a gate reading all fifty packets rather than one could, and it is red-proven in both directions on every invocation.

The result0 attack trials, and all four boundaries checked here hold on evidence — one of them only after a pre-flight gate convicted the code on it and it was fixed.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260822), so no text in it came from an outside party and there was nothing adversarial to resist. The corpus's own traps — a renumbering, a reused id, a scope exclusion and a prior note that something was “remediated after the period end” — test confusable PLAIN paperwork, which is a different thing from an adversarial author and must not be read as evidence about one.

HonestyWhat this does not prove

• Whether an indirect prompt injection in a Packet Notes section or a control description could move a change call, drop a row or flip the review flag. No attack run was fired, so the answer is unknown rather than negative.
• Whether the never-sent guard holds on a forker's own packets. It is asserted over these 50 and over these section names; a packet whose confidential material sits in a section not listed in NEVER_SENT is not protected by anything.
• Whether a real assurance report could be sent at all. Nothing here has considered whether a third party's assurance report is contractually permitted to leave your network, which on most vendor agreements is the first question and is not a technical one.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

needs_recert_review fires when the packet carries at least one control whose `change_kind` is `control_dropped` — a control the PRIOR report tested that the CURRENT report does not carry, with no scope change to explain it.

src/compare.py::compute(), called by compare() on every packet, by the local app on every click, and by evals/baseline.py on the free floor's own output so the two are judged by the same rule.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT a recertification, an approval, a suspension, a termination, a renewal, a finding or a clearance. Nothing in this kit writes, approves, notifies or dispatches anything; src/compare.py returns a JSON body and src/app.py serves one.
• It is NOT a check on the model. That is a different pure-code step — the CONFIDENCE FLOOR in src/compare.py::hold(), which re-runs the rulebook over each row's own stated facts and holds the row where they disagree. It held 0 rows on this run, which is what you would expect from a run with no wrong calls in it, and that zero is therefore evidence of nothing.
• It is NOT a materiality judgement. A dropped access control and a dropped visitor-escort control fire it identically. It knows nothing about what the service does, how much of it the control covers, what the contract says or what compensating controls exist.
• It is NOT complete. It fires on ONE of the eight change kinds. A packet whose opinion went from unqualified to disclaimer with every control unchanged does not fire it, and on this corpus 11 packets have a worsened opinion.

WatchedWhat is watched, and why that one

The kit measures 22 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 15 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (documents 50, extraction_cells 4635, failures 0, refusal_cells 0, thinking True) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

3 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked.

DeviationsWhat deviated

0 breaches across 3 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the flag's evidence on any change to compute() in src/compare.py, on any change to data/rulebook.json (which moves the change calls the flag reads), on any change to the corpus generator's dealt quotas (which move the band), and on any model or provider change. evals/check_labels.py re-measures the band and the selection guard for free on every invocation and refuses to let a run spend if either has drifted.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt is empty on purpose: the corpus is generated in-process, the rulebook is a JSON file read with json, the aligner is a set operation, the provider is reached over urllib and the scorer is exact match. There was nothing left to add. The cost of the position is stated below rather than waved away.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

The ANSWER has a graph in it — two control sets and the edges between them — but the PIPELINE does not. One path per packet: segment, align, select, prompt, one call, parse, hold, compute. No branch, no loop, no retry above the transport layer and no state between packets, which is why evals/run.py can run ten of them concurrently with a thread pool and nothing else.

The other sideWhat a framework costs you

• Everything is hand-rolled, so everything is yours to maintain — the JSON extraction from a fenced reply, the retry classification, the section cutter and the span locator. Four of those have each cost a sibling kit in this series a real defect.
• No observability beyond what the kit prints. There are no traces, no spans and no dashboard; what exists is the result file, and it exists because evals/run.py writes it.
• No structured-output guarantee. src/prompt.py::parse fails CLOSED to {} rather than to a plausible-looking empty worksheet, which is the right failure — but a schema-enforcing API would make the failure impossible instead of survivable.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-vendor-recert on the fast tier, 2026-08-22. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

Movement is against the most recent comparable run — same model tier, same guards. A run on a different tier is a different system and is never differenced, which is why some rows show a dash.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

1 run not plotted. b000-vendor-recert-idjoin recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — a PAIR is the unit; alignment is computed per pair.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT -- this repository's own licence. Every vendor, service, audit firm, person, site, telephone number, email address, contract, fee, control, control id, exception and opinion is invented; no real third party, service auditor, assurance report or recertification file is reproduced or referred to, and no assurance standard, attestation framework, regulator, supervisory body or industry group is named. The rulebook shipped with this kit is ILLUSTRATIVE and is not an authority. It was written for this kit and reproduces no assurance standard, attestation framework, regulator's third-party risk guidance, supervisory expectation, industry body guidance or firm's own vendor recertification procedure, and it names none of them. The criteria set the corpus cites (ARC-4) does not exist. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one line⚑ The join — one control, one row, however the two reports number it

Did the worksheet carry exactly the controls the union of the two reports contains? Micro precision and recall over control_ref, with the two directions named and never averaged: a MISSED control is one in the union and left off, a PHANTOM is a row for a control that is in neither report — which on this corpus is almost always the PRIOR id of a renumbered control returned as a row of its own. Split two ways, because a single percentage is not an answer: RENUMBERED against SAME-ID, and BY CHANGE KIND, because a missed no_change row is untidy and a missed control_dropped row is a control nobody noticed had stopped being covered.

MethodHow the test was run

evals/judge.py::score_join, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, which records every control in the union of the two reports under one key — the CURRENT id where the control appears this year and the PRIOR id where it does not. evals/check_labels.py asserts the key is unique within every packet and that each one heads a real section in its own file.

How tight can the band be? No tolerance band — a control is on the worksheet or it is not, matched on the id with whitespace collapsed and case folded. That is the ONLY normalisation, and it is deliberate: it forgives c-012 for C-012 and it does not forgive C-12, which would be a different control. A duplicate id is kept under a suffixed key rather than overwriting, because returning one control as two rows is the commonest wrong answer here and a dict that silently merged them would score it as a clean hit.

Cadence: Re-score on any change to src/align.py, to the id schemes in tools/build_corpus.py, or to the renumbering quota.

The decisionWhen to reach for it

In one lineThe renumbering — did it carry last year's id across

Of the 100 rows the two reports number differently, how many carry the right prior_ref? This is the one cell that proves the join was MADE rather than guessed, and it is scored on its own because it is a small number of rows carrying most of the argument. The mirror is scored too: on the 505 rows that keep their id, prior_ref must be the literal none, and a run that filled it in would be inventing a renumbering.

MethodHow the test was run

evals/judge.py::score_join's renumbering block, in-process, no key.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: gold's prior_ref, written by tools/build_corpus.py when it assigns the incoming auditor's numbering scheme, and asserted by evals/check_labels.py to head a real Prior Control section and never to equal control_ref.

How tight can the band be? Exact match after the same case-and-whitespace fold the join uses. none is compared as a literal string, not as a null.

Cadence: Re-score on any change to the id schemes or the auditor-change quota in tools/build_corpus.py, or to rule 2 of src/prompt.py.

The decisionWhen to reach for it

In one linePer-cell exact match, over every gold row

Is every value on the worksheet the value the two reports state? 4,635 cells: seven fields on each of 605 control rows, and eight packet-level fields on each of 50 packets. THIS IS THE REFERENCE GRADER — everything else on this page is a judgement about the comparison, and this is whether the readings under it are right. ⚑ THE DENOMINATOR IS GOLD'S ROWS, NOT THE RUN'S: a gold row the run never returned scores as a miss on all seven of its cells rather than disappearing, so a run cannot raise this number by comparing fewer controls.

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, which holds the value the generator WROTE into each report — and for the two derived fields, the shipped rulebook's own lookup over the three stated facts rather than a typed label. evals/check_labels.py re-runs that lookup over all 605 rows before any run may spend.

How tight can the band be? Exact match after light normalisation — trailing punctuation stripped, whitespace collapsed, case folded. No numeric tolerance, because every field on this kit is an identifier or a closed enum and there is no float anywhere in the reply. Case folding forgives Repeat_Exception for repeat_exception and forgives nothing else.

Cadence: Re-score on any change to data/fields.json, to the section layout in tools/build_corpus.py, or to the normalisation at the top of evals/judge.py.

The decisionWhen to reach for it

In one lineThe change call — eight ways, on the rows both sides list

The eight-way change call, on the controls BOTH sides list. Scored on matched rows only, deliberately: a call about a control the run never found is a JOIN failure, and counting it twice would let one mistake sink two graders. The full confusion matrix is in the run record, and it is the thing to read rather than the percentage — the eight values are not equally costly and repeat_exception mistaken for new_exception is the mistake this kit exists to prevent.

MethodHow the test was run

evals/judge.py::score_calls, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the shipped rulebook's own lookup (src/rulebook.py::change_kind) over gold's three stated facts — not a typed label. One definition, three readers: the corpus generator writes gold with it, src/prompt.py states it to the model in words, and evals/judge.py re-runs it for the no-gold consistency diagnostic.

How tight can the band be? Exact match on a closed eight-value enum after case folding. There is no partial credit and no ordering between the values, because control_dropped is not 'nearly' no_change.

Cadence: Re-score on any change to data/rulebook.json — which moves the rule, the model's instructions and the gold labels together — or to the dealt shape quotas in tools/build_corpus.py.

The decisionWhen to reach for it

In one lineThe opinion call — five ways, once per packet

The five-way opinion movement, once per packet, from the two reports' own opinions and the rulebook's four-value rank. Asked independently of every control row: a report can carry an unchanged opinion over a control set that moved underneath it, and that combination is exactly what a recertification is looking for.

MethodHow the test was run

evals/judge.py::score_calls's opinion_call block, in-process, no key.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/rulebook.py::opinion_movement over gold's prior_report_present, prior_opinion and current_opinion, re-derived by evals/check_labels.py on every packet.

How tight can the band be? Exact match on a closed five-value enum after case folding. The rank table (unqualified 0, qualified 1, adverse 2, disclaimer 3) is data in data/rulebook.json, not code, so a firm that ranks a disclaimer differently edits one file.

Cadence: Re-score on any change to the opinion rule or the rank table in data/rulebook.json.

The decisionWhen to reach for it

In one lineOver-confidence — a change asserted where the reports do not join

Of the 72 rows gold says the two reports do NOT let you compare, how many did the run answer with a definite change anyway? On a recertification worksheet this is the expensive direction: a change recorded against a control whose two ends do not join is a finding nobody can check, and no_change recorded against one is worse, because it says somebody looked. Split by WHY the row is open — a scope exclusion, a reused id and a first-cycle vendor are three different mistakes with three different fixes. The mirror, overcautious, is counted beside it and does not net against it. The score below is the share DECLINED, so higher is better.

MethodHow the test was run

evals/judge.py::score_calls's determinacy block, in-process, no key.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: gold's comparability, dealt in counted quantities by tools/build_corpus.py — 19 scope_withdrawn, 45 no_prior_report, 8 id_reused — with a floor asserted for each by evals/check_labels.py so no reason is anecdotal.

How tight can the band be? A row counts as over-confident when gold says not_comparable and the run returned any of the other seven values. There is no partial credit: no_change on a row nobody could compare is counted exactly as heavily as new_exception, because both assert the comparison was made.

Cadence: Re-score on any change to the comparability quotas in tools/build_corpus.py or to step 1 of the change rule in data/rulebook.json.

The decisionWhen to reach for it

In one lineThe review flag, against gold

The pure-code business flag, computed from whatever the run returned, against the same rule computed over gold. Recall and precision separately and never averaged: a missed escalation is a control nobody chases, and a false alarm is a reviewer sent back to a vendor for nothing. It is a check on the BUSINESS CONDITION, not on the model — that is the confidence floor, which is a different pure-code step.

MethodHow the test was run

evals/judge.py::score_flag, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/compare.py::compute() run over GOLD's rows, so the same rule grades both sides. The condition was chosen against a measured table of five candidates printed by tools/build_corpus.py, and evals/check_labels.py fails the build if its firing rate leaves a 10-60 pct band.

How tight can the band be? Boolean, no threshold. It fires on the presence of at least one row whose change_kind is control_dropped and returns None — not False — when the reply carries no row the rule can read, because an unknown is not a pass.

Cadence: Re-check on any change to compute() in src/compare.py, to data/rulebook.json (which moves the calls the flag reads), or to the dealt quotas that set the band.

The decisionWhen to reach for it