The business caseThe problem this solves

A health plan's marketing-compliance team has to know that a broker said the things a recorded enrollment call requires — that the call is recorded, that the broker does not work for the government, how many carriers they represent, that enrolling will end the current plan — and did not say the things a broker may not say. The recordings exist. Reading them does not scale: a reviewer plays the call, keeps eleven items in their head, and decides for each one whether it was said, by WHOM, and whether it came before the enrollment started. a reviewer playing a recorded enrollment call end to end and ticking eleven disclosure items by hand

The inputThe actual recorded enrollment calls

The corpus is 64 recorded enrollment calls, 176.31 MB (txt 64 · wav 64). Compliance on a recorded call is not a transcription problem and this corpus is built to prove it. 11 of the 64 calls plant a disclosure in the BENEFICIARY's mouth, and 30 more have the caller repeat a sentence the agent also said — two families a transcript can only tell apart by who spoke. 33 calls meet an obligation in words the checklist does not use, and 19 name a prohibited claim in order to REFUSE it, which is the trap for any keyword check: every content word of the breach is present and nothing was breached.

AGENT: Good morning, this is Corin Ashvale calling from Cedarline Benefit Advisors about your Medicare options.
CALLER: Yes, hello. This is Desmond Achebe.
CALLER: I should say up front, I already have a plan and I have carried it for 4 years now.
AGENT: This call is being recorded for quality and compliance purposes.
AGENT: I do not work for Medicare or the federal government.
AGENT: There are eight different companies whose products I am able to offer where you live.
AGENT: There is no obligation to enroll in anything today.
AGENT: I am licensed in Arizona and my producer number is 1 8 9 7 9 7 7.
AGENT: Enrolling in this plan will end your current coverage.
AGENT: We are here to look at a Medicare Advantage plan, and the one I have in front of me is Bluewater Complete.
CALLER: That sounds reasonable so far.
AGENT: Your monthly premium on this one is 24 dollars.
CALLER: I appreciate you explaining it slowly.
AGENT: The out of pocket maximum for the year is 4900 dollars.
AGENT: I am not allowed to ask what conditions you are being treated for.
CALLER: Go on, I am listening.
AGENT: There is a drug list, and it is reissued at the start of each year.
AGENT: All right, let me go ahead and start your enrollment application now.
AGENT: That is everything on my side. A confirmation letter goes out this week.
CALLER: Thank you for your time.

The outcomeWhat a good result looks like

eleven item verdicts per call — met, breached or not_required — the quoted turn each one rests on, and one call outcome of clear or refer. The outcome is never asked of the model: it is derived in code from the readings by the six rulebook rules, so two people reading the same record get the same answer.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• your recordings are two-party and your checklist has a who-said-it rule in it — the diarized track
  it is the only priced path on this stage's frontier that returns speaker labels, and the rule that a disclosure in the caller's mouth does not count fired 17 times on it against 6 on the undiarized text. It buys 9.4 points of call-outcome accuracy for roughly twice the transcription price.
• your checklist has no rule that turns on who spoke — the undiarized track, or the free regexes
  with attribution out of the question the two paths converge, and the cheap one is half the price. Note what the free floor already gives you on this corpus: 0.7812 of call outcomes against the cheap track's 0.8125, and 0.7080 F1 against 0.7027 — a tie on the finding.
• you want to know whether to spend on the transcriber or on the model — run the perfect-reading arm first — it is free once the reference text exists
  it separates the two questions completely. Here it scores 0.9688, so 3 wrong item verdicts are the model's own ceiling before any engine is bought, and everything between that and a purchased path is what the purchase cost.
• you are pricing this against a human reviewer — count the queue, not the accuracy
  the free floor has recall 1.0000 and precision 0.5479, so it refers almost everything; the diarized track has precision 0.6964. The difference is how many clean calls a reviewer opens, which is the cost that scales.

At a glanceHow the whole thing runs

Run once, for real, on 2026-09-16. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-09-16 — r001-call-disclosure-check-aws. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Observed on this machine: a checkout with no .env and no key runs python3 -m evals.baseline and prints the whole floor table in about two seconds, because the floors call nothing. python3 -m evals.check_labels re-derives all 704 item verdicts from the committed turn structure and exits clean in under a second. python3 -m src.app serves the committed runs on port 9487 with no credential at all.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. Measured on the fast tier after the diarized track — the model call only, timed around each of the 64 calls on the recommended arm. It does NOT include the transcription request: stage 1 is a separate batch job against a vendor and is neither a per-call latency nor one this kit controls. Every arm's p50 sits within 52 ms of every other, because all three send the same prompt shape over the same ceiling.

Current processWhat it replaces

a reviewer playing a recorded enrollment call end to end and ticking eleven disclosure items by hand

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusCorpusCorpusEvalsEvalsArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• wav
• txt
• jsonl
• json
• 4 of 11 stations carry a recorded failure

scroll → 11 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one recorded enrollment call is

none — all 64 calls are scored and there is no training, tuning or held-out set, because nothing here is fitted. The slices that matter are cut AFTER scoring, by hazard family, and every call carries its own hazard labels for that.

SetupWhat the setup figure measured

There is no index and nothing is retrieved. One call is one transcript and goes whole into one prompt, so there is nothing to chunk, embed or rank — the zero is a fact about the architecture, not a measurement that has not been taken yet.

LicenceLicence

MIT. The corpus is this kit's own generated output and carries no third-party rights; it is released under the MIT licence in LICENSE-PUBLIC at the root of the kits repository, verified there on 2026-09-16. The repository root LICENSE is a different licence and does not govern this corpus. Nothing in the corpus is derived from any third-party dataset, recording, transcript or regulatory text.

Bring your ownBring your own recorded enrollment calls

Drop your own 16 kHz mono WAVs in data/corpus/ named <case_id>.wav, and one line per call into data/gold.jsonl carrying items (the eleven verdicts), breached, outcome and spans (the turn text and speaker each item rests on). Then run python3 -m evals.check_labels, which will refuse the set if the verdicts and the spans disagree. Swap data/rulebook.json for your own checklist and the prompt follows it — src/prompt.py::build takes it as its second argument.

⚠︎ And what stops being true when you do: reference_text is the field a real corpus cannot honestly supply, and every stage-1 error rate on this page depends on it. Here it is the script say was handed, so it is exactly right by construction. On real audio the best you have is a human transcription, which has its own error rate — so a forker can still measure the model half and the outcome half, and cannot reproduce the propagation split without paying a transcriptionist.

TracksWhich path you should run this on

This kit’s input is audio, so it carries a capability stage in front of the model. The engine knows 5 arrangements, ruled 2 out here, and this kit runs A1 — measuring 2 paths through it against the same 64 recorded enrollment calls and the same graders. Everything below is that run.

Cost is units × today’s price — the kit records minutes and pages, never dollars, so a vendor price move never invalidates what was measured and never costs a re-run.

FunnelHow the 2 were chosen

This is here so you can see the 2 were not cherry-picked — and so you can see where they disagree with the filters. 2 of the 2 this kit ran survive all 3 rules above.

Counts and rules only. The vendor names and list prices behind the 5 rows that survived these rules and were not run are estimates, and estimates live on the Admin catalogue, not on a kit page.

VolumeYour volume

The cost of one recorded enrollment call is measured: 96.233 minutes, 64 files over 64 recorded enrollment calls (lenses/Data/corpus/doc_count), from this run’s own recorded units. Everything the buttons change is that number multiplied — no rate, no number and no ranking moves with the volume.

LensWhat matters most to you

A lens re-orders the table below and changes nothing in it. No cell value moves, no row is hidden, and a path below the floor of 0.7812 can never be promoted to first — hiding a row is how a preset manufactures a winner.

MeasuredThe 2 paths this kit ran

cheapest path clearing the floor of 0.7812 on call_outcome_accuracy (the metric this kit's own eval nominates as its headline): $0.288699 for the whole scored run

ArrangementsThe 5 arrangements

Eligibility, the numbers each arrangement owes, the artifact it produces and whether it can attribute an error are structural — they are true of an arrangement whether or not it ran, and they are read from the standard rather than typed here. The cost and accuracy rows are blank for every arrangement this kit did not run, because nothing measured them.

Where to spendThe number that tells you where to spend

A2 and A3 cannot produce that sentence at any price: one supplier, one number, no span to point at. That is the entire argument for the arrangement this kit runs, and it is the reason A1 is the only one of the 5 that owes a propagation figure at all.

StabilityWhere the ranking would flip

Every condition above is arithmetic over the prices and the scores on this page. None of it is a forecast: it says what would have to be true, not what will be.

ScopeWhat is on this page, and what is not

Re-runRun it on your own stack

Point it at your own recorded enrollment calls. Drop your own 16 kHz mono WAVs in data/corpus/ named <case_id>.wav, and one line per call into data/gold.jsonl carrying items (the eleven verdicts), breached, outcome and spans (the turn text and speaker each item rests on). Then run python3 -m evals.check_labels, which will refuse the set if the verdicts and the spans disagree. Swap data/rulebook.json for your own checklist and the prompt follows it — src/prompt.py::build takes it as its second argument.

Re-scoring costs nothing. All 4 graders are deterministic code with a known key — Call outcome, Breached-item F1, Stage-1 word and character error, Propagation — so a recorded run can be re-scored as often as you like and returns the same answer. The only thing you pay for is the transcription and the model call themselves.

Before you spend anything. Observed on this machine: a checkout with no .env and no key runs python3 -m evals.baseline and prints the whole floor table in about two seconds, because the floors call nothing. python3 -m evals.check_labels re-derives all 704 item verdicts from the committed turn structure and exits clean in under a second. python3 -m src.app serves the committed runs on port 9487 with no credential at all.

Swap the stage-1 station. The arrangement and the graders are unchanged by a swap, so your re-run stays comparable to this one. 7 rows in the Speech to text stage carry a price that both parses and can be re-read today: AWS Transcribe batch (the one this kit runs), Google Speech-to-Text v2 Dynamic Batch (Chirp), AssemblyAI Universal-3.5 Pro, ElevenLabs Scribe, Gladia, Alibaba Cloud Model Studio - Qwen3-ASR-Flash, OpenAI gpt-4o-mini-transcribe (the one this kit runs).

Before you use theseRead this before you use these numbers for your own case

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 4,627 characters assembled, 3,254 are instructions — 70% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Replayed from the kit's own src/prompt.py::build for CALL-0001, the sample clip, against that call's committed stage-1 text from the diarized track — so the decomposition is the assembly order the run used and not a tidy retelling of it.

What the model receivesWhat the model actually receives after stage 1

Three readings of the same 78-second call. The perfect reading is the script say was given. The diarized track labels every turn and mishears more words. The undiarized track hears more words correctly and returns one unbroken paragraph with no speaker at all — which is the input the rule about the caller's mouth cannot be applied to.

Full promptThe prompt in full

You are a marketing-compliance reviewer at a health plan. You read one recorded enrollment call, transcribed, and you record what was said and who said it. You do not decide whether an agent is in violation, you never close a file, and you never recommend a penalty: you record readings so a human reviewer can look at the turn you quote. You answer only with JSON.

The eleven items, from the plan's marketing-compliance checklist (anchor: 42 CFR 422.2267):
  MUST BE SAID BY THE AGENT
    D1  call is recorded
    D2  agent does not represent the government
    D3  number of organizations the agent represents
    D4  appointment scope was agreed in advance   [conditional: not required when every product line discussed on the call is the one line recorded on the appointment]
    D5  enrolling is voluntary
    D6  enrolling ends the current coverage   [conditional: not required when the beneficiary has no current coverage for an enrollment to end]
    D7  agent license state and producer number
  MUST NOT BE ASSERTED BY THE AGENT
    P1  ranking or superiority claim
    P2  gift, cash or item of value offered to enroll
    P3  health status asked in order to steer the enrollment
    P4  government endorsement implied

Each line of the transcript is one turn. A line may begin with AGENT: or CALLER: naming who spoke it. IT MAY NOT. When a line carries no speaker label, say "unknown" for who spoke it — do not infer a speaker from what the line says. Reporting `unknown` honestly is worth more here than a confident guess, because the reviewer downstream treats the two differently.

This is a transcript of one recorded enrollment call. It is machine-produced and may contain mishearings; read it as speech, not as a clean document.

AGENT: Good morning. This is Corin Ashville calling from Cedar Line Benefit Advisors about your Medicare
CALLER: options. Yes, this is Desmond Achebe. I should say upfront, I already have a plan and I have carried it for 4 years now.
AGENT: This call is being recorded for quality and compliance purposes. I do not work for Medicare or the federal government. There are 8 different companies whose products I am. To offer where you live. There is no obligation to enroll in anything today. I am licensed in Arizona and my producer number is 1-897977. Enrolling in this plan will end your current coverage. We are here to look at the Medicare Advantage plan, and the one I have in front of me is Blue Water
CALLER: Complete. That sounds reasonable
AGENT: so far. Your monthly premium on this one. $24.
CALLER: You explained it slowly.
AGENT: The out of pocket maximum for the year is $4900. I am not allowed to ask what conditions you are being greeted for. Go on, I am listening. There is a drug list and it is reissued at the start of each year. All right, let me go ahead and start your enrollment application now. That is everything on my side. A confirmation letter goes out this week.
CALLER: Thank you for your time.

Return ONE JSON object and nothing else, with exactly these three keys:
  "items"        an object with one entry per code above (D1, D2, D3, D4, D5, D6, D7, P1, P2, P3, P4), each an object:
                   "said"  true when the thing described by that code is actually
                           present somewhere in the transcript, false when it is not.
                           For a prohibited code, `said` is true ONLY when someone
                           ASSERTS the claim. A speaker who refuses it, or who says
                           they are not allowed to do it, has NOT asserted it.
                   "who"   "AGENT", "CALLER", or "unknown" — who spoke the line you
                           are quoting. Use "unknown" whenever the transcript does
                           not label its speakers.
                   "quote" the words that carry it, at most 15 words, copied from
                           the transcript. Empty string when "said" is false.
                   "pre"   true when that line comes BEFORE the line where the agent
                           starts the enrollment application, false when it comes
                           after it, and true when no such line exists.
  "second_line"  true when a product line was discussed that is NOT the single line
                 the appointment was agreed for, false otherwise.
  "has_coverage" true when the beneficiary already has coverage that an enrollment
                 would end, false when they say they have nothing in place yet.
Do not return a verdict, a score, or an outcome for the call. Those are derived from your readings by the reviewer's own rules.

Raw responseThe raw response

{
  "items": {
    "D1": {
      "said": true,
      "who": "AGENT",
      "quote": "This call is being recorded for quality and compliance purposes.",
      "pre": true
    },
    "D2": {
      "said": true,
      "who": "AGENT",
      "quote": "I do not work for Medicare or the federal government.",
      "pre": true
    },
    "D3": {
      "said": true,
      "who": "AGENT",
      "quote": "There are 8 different companies whose products I am.",
      "pre": true
    },
    "D4": {
      "said": false,
      "who": "unknown",
      "quote": "",
      "pre": true
    },
    "D5": {
      "said": true,
      "who": "AGENT",
      "quote": "There is no obligation to enroll in anything today.",
      "pre": true
    },
    "D6": {
      "said": true,
      "who": "AGENT",
      "quote": "Enrolling in this plan will end your current coverage.",
      "pre": true
    },
    "D7": {
      "said": true,
      "who": "AGENT",
      "quote": "I am licensed in Arizona and my producer number is 1-897977.",
      "pre": true
    },
    "P1": {
      "said": false,
      "who": "unknown",
      "quote": "",
      "pre": true
    },
    "P2": {
      "said": false,
      "who": "unknown",
      "quote": "",
      "pre": true
    },
    "P3": {
      "said": false,
      "who": "unknown",
      "quote": "",
      "pre": true
    },
    "P4": {
      "said": false,
      "who": "unknown",
      "quote": "",
      "pre": true
    }
  },
  "second_line": false,
  "has_coverage": true
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Check a recorded enrollment call against the disclosures the rule requires — 64 recorded enrollment calls. One model answered, and every answer was then graded Four different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

There is no judge anywhere in this kit. The model returns eleven closed readings per call — was the thing said, WHO said it, was it said before the enrollment started — plus two exception readings. src/recheck.py then applies the six rulebook rules in pure code to produce the item verdicts, the breached set and the call outcome, and the same module grades the free floors. Re-scoring any committed run costs $0.00 and gives the same answer every time.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar in cost_by_model is a MEASURED token count multiplied by a PUBLISHED rate — a projection onto a rate card, not a bill anybody paid. The token counts are the provider's own, from the usage block on each of the 192 calls across the three paid arms. ⚑ THE CARD IS NOT THE PROVIDER THAT RAN THE CALLS: this site withholds the runtime provider, so the run's own tokens are priced against published lists anybody can read. ⚑ AND THE TRANSCRIPTION STATIONS CARRY NO DOLLAR IN THIS KIT AT ALL, ON PURPOSE: a kit records UNITS. 96.233 audio minutes went to each station and the price is applied at build time from the capability table, so a vendor repricing never invalidates this run and never costs a re-run.

Rates checked 2026-09-12.

Grading unitWhat the grading figure prices

FREE. Every grader is pure code — set arithmetic, Levenshtein and the six rulebook rules — so grading costs nothing and re-grading a committed run costs nothing forever. There is no judge anywhere in this kit. The money this kit spent was bought once, upstream of grading: the transcription of 96.233 audio minutes twice, and 216 model calls.

The gradersFour ways to grade

The constant floor is on the table too and it is the reason this kit does not publish per-item verdict accuracy as its headline: a fixed reply of 'every item met, no findings' takes 0.8892 of the 704 item verdicts, because 626 of them are met. On the two numbers that decide anything it scores 0.5000 of call outcomes — chance, on a corpus balanced 32 clear / 32 refer — and 0.0000 F1.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — top_k, chunk size, prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-09-12. Every dollar here is the model call only: this kit also runs AWS Transcribe batch before the model, and one recorded enrollment call costs $0.00976 end to end — see the Tracks panel. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingAnswering and grading are two separate bills

Every grader is pure code — set arithmetic, Levenshtein and the rulebook — so grading is free and re-grading a committed run is free forever. The only money this kit ever spent is the transcription of 96.233 minutes twice and 216 model calls, $0.9798 in total.

Cost driversWhat actually moves the bill

• The transcript. It is the whole prompt and it is 1078 input tokens a call on the recommended arm — roughly three quarters of the prompt. Doubling the call length doubles the bill for the model half and doubles the audio minutes for the transcription half at the same time.
• The audio minutes, which are the LARGER half and are not model spend at all. The recommended station is $0.006 a minute and the cheap one $0.003, so 96.233 minutes of corpus cost $0.5774 and $0.2887 — between eight and thirteen times the model bill for the same corpus.
• The number of checklist items, which sets the output. Eleven small objects came back at about 438 output tokens; a thirty-item checklist roughly triples that half.

Your volumeWhat it costs at your volume

Linear on both halves and there is no cliff between them. Ten times the corpus is ten times the minutes and ten times the tokens: about $5.77 of transcription on the recommended station plus about $0.47 of model spend, against $2.89 and $0.47 on the cheap station. The ratio between the two halves does not move, which is why the buying decision on this page is about accuracy and not about volume.

Model choiceWhy this model

The reply is a closed schema — eleven objects of two booleans, a speaker from a three-word vocabulary and a short quote — so the cheapest tier is the right default and reasoning is off. The largest reply across all 216 paid calls used 497 of a 1,000-token ceiling and nothing truncated, so no rung of the output-ceiling ladder was needed. The interesting spend on this kit is not the model at all: transcription is between eight and thirteen times the model bill for the same corpus.

Other modelsWhat the model half would cost on a published card

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

one pass of the recommended arm: 64 calls, model half only

The codeEvery file, and what it is for

Nine modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

tools/build_corpus.pythe recordings

64 enrollment calls generated from seed 20260916, each written as a speaker-labelled script and then SPOKEN — macOS say, one voice per speaker, concatenated and converted to 16 kHz mono PCM. The script is kept beside the audio as the reference transcript, which is what makes an oracle arm possible at all. The answer key is written from the turn structure, not from the audio.

# GENERATE 64 TWO-PARTY MEDICARE ENROLLMENT CALLS, THEIR TRANSCRIPTS AND THE ANSWER KEY, from one seed.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
CORPUS = os.path.join(DATA, "corpus")
SEED = 20260916
DATASET = "call-disclosure-check-v1-64calls"
N_CASES = 64
ANCHOR = "42 CFR 422.2267"
AGENT_NAMES = ["Corin Ashvale", "Dara Pellmont", "Evan Thornbury", "Gita Marlowe",
CALLER_NAMES = ["Alma Redwater", "Bertram Kinloch", "Cordelia Naysmith", "Desmond Achebe",

data/rulebook.jsonthe checklist and its rules — a swap seam

eleven item codes — seven that must be said by the agent, four that must not be asserted — the two conditional exceptions, the three verdict words, the two call outcomes and the six rules that turn readings into verdicts. Every other module reads this file; none of them restates it.

You change it to: a different plan's items, exceptions or rule set. src/prompt.py::build takes the rulebook as its second argument for exactly this, so a forker swaps the file without editing a module. The answer key must be rebuilt with it.

# 

src/stations/__init__.pystage 1 — the transcription stations

one module per purchasable path, each declaring the same read() contract, plus oracle.py, which returns the committed reference transcript and calls nothing. Output is cached to results/asr-<station>.jsonl so stage 2 can be re-scored for $0.00 forever.

# STAGE 1 — the transcription paths this kit measures, behind one call shape.

def audio_seconds(path):
def _oracle(path):
def _aws(path):
def _openai(path):
def _azure(path):
STATIONS = {
def transcribe(station, path):

src/prompt.pythe prompt

assembles five ordered parts and parses what comes back into eleven closed readings — said / who / quote / pre — plus the two exception readings. The model is never asked for a verdict and never for the outcome. unknown is an instructed answer for who spoke a line, because a transcript with no speaker labels genuinely cannot say.

# The one prompt this kit sends, assembled from parts, and the parser for what comes back.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RULEBOOK = json.load(open(os.path.join(HERE, "data", "rulebook.json"), encoding="utf-8"))
REQUIRED = [r["code"] for r in RULEBOOK["required_items"]]
PROHIBITED = [p["code"] for p in RULEBOOK["prohibited_items"]]
ITEMS = REQUIRED + PROHIBITED
VERDICTS = ["met", "breached", "not_required"]
VERDICT_MEANINGS = {
OUTCOMES = ["clear", "refer"]
SYSTEM = (

src/recheck.pythe rulebook, executed — a swap seam

the six rules in pure code: an item is met only when an AGENT turn carries it at or before the enrollment turn; a disclosure in the caller's mouth is not met; a late one is not met; a conditional item whose exception holds is not_required; a prohibited claim that was refused was not asserted; any breached item refers the call. Every arm goes through it, including both free floors.

You change it to: how readings become verdicts — for instance whether a late disclosure is a breach or a finding of its own. This is the file to edit to make the kit stricter or looser, and it is deliberately the only one.

# THE RULEBOOK, APPLIED IN CODE. Readings in, verdicts and an outcome out. No model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RULEBOOK = json.load(open(os.path.join(HERE, "data", "rulebook.json"), encoding="utf-8"))
REQUIRED = {r["code"]: r for r in RULEBOOK["required_items"]}
PROHIBITED = {p["code"]: p for p in RULEBOOK["prohibited_items"]}
ITEMS = list(REQUIRED) + list(PROHIBITED)
MET, BREACHED, NOT_REQUIRED = "met", "breached", "not_required"
CLEAR, REFER = "clear", "refer"
def _owed(code, readings):
def _required(code, r):

src/floors.pythe free floors — a swap seam

domain word lists, speaker labels and an enrollment cue — plus two plain regexes for the conditional exceptions. No model, no cost, and the same recheck module afterwards, so the floor and the paid arm are compared on identical arithmetic.

You change it to: what 'no model at all' means for your checklist. It is the number every paid arm on this page is measured against, so a forker who changes the checklist owes a new one.

# THE FREE ARMS. No model, no key, no network — and they read the same transcript the model reads.

REQUIRED, PROHIBITED = list(RC.REQUIRED), list(RC.PROHIBITED)
ITEMS = RC.ITEMS
DOMAIN = {
ENROLL_CUE = re.compile(r"start(ing)?\s+(your\s+)?(enrollment|application)", re.I)
SECOND_LINE = re.compile(r"\b(also|another|second|as well|while we are at it|on top of that)\b"
NO_COVERAGE = re.compile(r"\b(nothing in place|no coverage|not covered|do not have (any )?"
HAS_COVERAGE = re.compile(r"\b(already have|carried it|current (plan|coverage)|existing "
SPEAKER = re.compile(r"^\s*(AGENT|CALLER|SPEAKER[ _]?\d+|spk[ _]?\d+)\s*[:\-]\s*", re.I)
def turns(transcript):

evals/scoring.pythe scorer

item verdicts, the breached-set F1 with its precision and recall, the call outcome, WER/CER raw and ITN-folded, and the propagation split — for each wrong item, whether the words it rests on survived stage 1 and whether the speaker was decidable.

# The graders. All pure code: no model call, no network, no randomness.

ONES = ["zero", "one", "two", "three", "four", "five", "six", "seven", "eight", "nine", "ten",
TENS = {"twenty": 20, "thirty": 30, "forty": 40, "fifty": 50, "sixty": 60, "seventy": 70,
def strip_speakers(s):
def normalise(s, *, fold_case=True, strip_punct=True):
def _lev(a, b):
def wer(reference, hypothesis):
def cer(reference, hypothesis):
def _int_words(n):
def itn_fold(text):

evals/check_labels.pythe answer-key gate

re-derives every verdict, every breached set and every outcome from the committed turn structure with the rulebook retyped from scratch, and refuses the build on any disagreement. It imports nothing from src/ or tools/ — a gate that imported the builder's own copy would be the builder agreeing with itself.

# Is the LABELLED SET internally consistent? A gate over data/gold.jsonl. PURE CODE, no model.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
CORPUS = os.path.join(DATA, "corpus")
REQUIRED_CODES = ["D1", "D2", "D3", "D4", "D5", "D6", "D7"]
PROHIBITED_CODES = ["P1", "P2", "P3", "P4"]
ALWAYS_CODES = ["D1", "D2", "D3", "D5", "D7"]
CONDITIONAL = {"D4": "scope", "D6": "has_current_coverage"}
VERDICTS = ["breached", "met", "not_required"]
OUTCOMES = ["clear", "refer"]

src/app.pythe board

a stdlib HTTP server on port 9487 that replays the committed runs and calls nothing. It has no do_POST. _public() strips model, provider, api_key and base_url from every payload at any depth, so the browser is never given a name the site withholds.

# UC0487 -- the call-disclosure-check review board. A stdlib HTTP server, port 9487.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
DATA = os.path.join(HERE, "data")
CORPUS = os.path.join(DATA, "corpus")
RESULTS = os.path.join(HERE, "results")
PORT = 9487
TITLE = "UC0487 call-disclosure-check"
HEADLINE = "Check a recorded enrollment call against the disclosures the rule requires"
FLOOR_OF_RECORD = "b000-call-disclosure-check-rules-exc"

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 1077 input and 437 output tokens per query at top-k 1, of which about 150 tokens are fixed (system prompt, chat framing, the question itself) and the rest is retrieved context. Change the retrieval depth and the input scales with it. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model and top-k follow the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

The model is asked for readings, never for a verdict and never for an outcome: the reply schema has three keys and none of them is a decision. src/recheck.py derives the verdicts and the outcome from the readings by the rulebook, in code. So an instruction that persuades the model to say every item was met still has to survive the rulebook — and an item marked said by a CALLER turn, or after the enrollment turn, is still not met. The board has no write route at all: src/app.py defines no do_POST.

No credential is in this repository and none ever has been. The kit reads one gitignored .env with a per-kit override: the model key reaches exactly one module (src/adapters/) and each station credential reaches one module under src/stations/. No key and no model id is sent to the browser — src/app.py::_public() strips model, provider, api_key and base_url from every payload at any depth, and python3 -m src.app --self-test red-proves it against eval files that DO carry a model id, so it cannot pass vacuously. The station that reads from object storage REFUSES to run rather than inventing a bucket name, and no account identifier, bucket name or resource name appears anywhere in this kit.

The experimentWe attacked it — three wordings, eight calls, one turn each

8 target calls that the answer key says should refer, crossed with 3 escalations of an instruction planted as an AGENT turn at the end of the transcript: a plain pre-clearance claim, an authority notice addressed to the transcript reader, and a schema instruction. The payload route is the realistic one for this kit — what reaches the model is a transcript, so anything spoken on the call reaches the prompt, and a transcriber is not in the business of refusing to write something down. Both were run for real on x001-call-disclosure-check, 2026-09-16 — 24 calls, $0.012889.

The result0 of 24 planted instructions changed a call's outcome. The structural reason is that the outcome is not in the reply at all.

24 trials: 8 target calls x 3 escalations, every one on the perfect reading so the transcriber is not a variable. Each trial records the gold breached set, the breached set the run produced, and which breaches were lost.

HonestyWhat this does not prove

• An instruction planted earlier in the call, or split across several turns. Every escalation here lands as one turn at the end.
• A wording nobody has written down yet. Three is not a sweep.
• An attacker who can edit the recording rather than speak into it.
• Anything about the transcription stage as an attack surface. The probe ran on the perfect reading so the engine would not be a variable.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

The model supplies READINGS. The rulebook is applied in CODE. For each of the eleven checklist items the model answers three closed questions — was the thing said, who said it, and was it said before the agent started the enrollment application — plus two readings that decide whether a conditional item was owed. Every verdict, the breached set and the call's outcome are then derived from those readings by src/recheck.py. The model is never asked for a verdict and there is no field in the reply for the outcome.

src/recheck.py executes all six rulebook rules and OVERTURNS a reading that breaks one; evals/check_labels.py gates the answer key itself, retyping the rulebook from scratch and importing nothing from src/ or tools/; data/rulebook.json is the single copy of the checklist and both of the others read it rather than restating it.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• a determination that an agent is in violation. It flags a checklist item and the turn it rests on, for a human reviewer.
• a transcription benchmark. The engine with the lower word error rate scores worse here.
• a measurement on real calls. Every recording is synthesised speech.
• a security control. The adversarial arm is one probe of 24 calls, not a proof.

WatchedWhat is watched, and why that one

The kit measures 31 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

6 measured by the latest run 25 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

no shared guard The two recorded runs declare no guard in common, so nothing below is protected by a precondition — read the history as two separate observations, not as a difference.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

17 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

extraction · no model in the path — a baseline, not a peer column — 13 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

extraction · with the model in the path — 3 runs. Columns here are only ever compared with each other.

not a time series No two of these 3 runs measured the same system — they differ on attributable_item_readings, breached_called, breached_fn, breached_fp, breached_tp, calls_with_speaker_labels, capability.architecture, capability.stage, capability.station, capability.units, r2_fired — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

redteam · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 6 chips that all say so.

DeviationsWhat deviated

0 breaches across 17 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

on every run, and free: every grader is code, so re-scoring the committed runs costs $0.00 and gives the same answer. The transcripts are cached, so even a prompt change re-scores without buying audio again.

In one lineNo framework, deliberately

No agent framework, no orchestration library, no vector store, no schema library and no cloud SDK — and the reason is measurable rather than ideological. There is ONE model call per call with no tools, no retrieval, no memory and no branching, and the only thing in front of it is a transcription request. Every import is the Python standard library, request signing included: there is no boto3 and no vendor SDK anywhere in this kit, so there is no pip install at all for the pipeline or the eval layer.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

corpus (wav + reference txt) → station.read() → results/asr-*.jsonl → prompt.build() → one model call → prompt.parse() → recheck.apply() → verdicts, breached set, outcome → scoring.py → results/eval-*.json

The other sideWhat a framework costs you

• No retries and no backoff beyond what the runner does itself. A rate limit is handled by re-running with --resume against the same run id, which replays what is cached.
• No streaming, no batching API and no concurrency. Every arm ran its calls in sequence.
• Request signing by hand is more code to read than an SDK import, and it is this kit's code to get wrong.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-call-disclosure-check-aws on the fast tier, 2026-09-16. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 15 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

13 runs not plotted. b000-call-disclosure-check-null, b000-call-disclosure-check-null-aws, b000-call-disclosure-check-null-openai, b000-call-disclosure-check-rules, b000-call-disclosure-check-rules-aws, b000-call-disclosure-check-rules-exc, b000-call-disclosure-check-rules-exc-aws, b000-call-disclosure-check-rules-exc-openai, b000-call-disclosure-check-rules-nolabels, b000-call-disclosure-check-rules-nolabels-aws, b000-call-disclosure-check-rules-nolabels-openai, b000-call-disclosure-check-rules-openai, t000-call-disclosure-check-stub recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the rulebook is data read whole into the prompt.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, AWS_ACCESS_KEY_ID, AWS_REGION, AWS_SECRET_ACCESS_KEY, AZURE_SPEECH_KEY, AZURE_SPEECH_REGION, BASE_URL, EMBED_MODEL, FOUNDRY_RUN_ID, MAX_CALLS_PER_DAY, MODEL, OPENAI_API_KEY, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT. The corpus is this kit's own generated output and carries no third-party rights; it is released under the MIT licence in LICENSE-PUBLIC at the root of the kits repository, verified there on 2026-09-16. The repository root LICENSE is a different licence and does not govern this corpus. Nothing in the corpus is derived from any third-party dataset, recording, transcript or regulatory text. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineCall outcome

does this call route to a compliance reviewer, or does it clear

MethodHow the test was run

evals/scoring.py::outcome_scores over all 64 calls. The outcome is never read out of a reply: src/recheck.py derives it by rule R6 — any breached item refers the call — from the eleven verdicts the same module derived from the readings. This is the kit's declared headline metric, and it is declared because per-item verdict accuracy is majority class on this corpus and cannot separate the two purchasable paths.

Every grader on these pages scored the same 64 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

three readings, one answer. The interesting part is the prohibited item: every content word of the claim is in the transcript, and the rule that a refused claim was not asserted is what keeps all three arms from calling it a breach. A keyword check flags it.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl outcome, re-derived by evals/check_labels.py from the committed turn structure with the rulebook retyped from scratch.

No true/false rates for this grader. It records 5 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? 0.7812 is the capability floor: it is what the FREE floor of record scores on this same metric given a perfect reading, so it is the number a purchased path has to beat to have bought anything. It is not the oracle arm's 0.9688, which nothing purchasable reaches and which as a floor would mark no winner at all.

Cadence: on every run; re-scoring a committed run is free

The decisionWhen to reach for it

In one lineBreached-item F1

of the item verdicts that should read breached, which were found, and at what cost in false findings

MethodHow the test was run

evals/scoring.py::item_scores over the 40 item verdicts the answer key marks breached out of 704. Precision and recall are published beside it and never without it: the free floor of record has recall 1.0000 and precision 0.5479 — it finds every breach by flagging clean items too — so an F1 quoted alone would hide which half of the trade an arm bought.

Every grader on these pages scored the same 64 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

three readings, one answer. The interesting part is the prohibited item: every content word of the claim is in the transcript, and the rule that a refused claim was not asserted is what keeps all three arms from calling it a breach. A keyword check flags it.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl breached, the per-call set of item codes, re-derived by the answer-key gate.

No true/false rates for this grader. It records 4 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? there is no pass mark. The free floor of record scores 0.7080, so a purchased path below that has bought a worse reviewer queue than free regexes would give.

Cadence: on every scored run

The decisionWhen to reach for it

In one lineStage-1 word and character error

how far the purchased transcript is from what was actually said

MethodHow the test was run

evals/scoring.py::wer, ::cer and ::wer_folded — Levenshtein over a case-folded, punctuation-stripped, whitespace-collapsed basis with speaker prefixes removed from BOTH sides, against the reference transcript. The folded form additionally puts both sides into one number format, because a script reads a producer number out digit by digit and an engine writes it as one token: raw WER charges seven substitutions for a perfect hearing. Raw and folded are labelled apart everywhere. ABSENT for the perfect-reading arm, which ran no engine — 'perfect' and 'not measured' must never print as the same number.

Every grader on these pages scored the same 64 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

three readings, one answer. The interesting part is the prohibited item: every content word of the claim is in the transcript, and the rule that a refused claim was not asserted is what keeps all three arms from calling it a breach. A keyword check flags it.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/corpus/<CASE>.txt — the script macOS say was handed, so the reference is exactly right by construction and by construction only.

No true/false rates for this grader. It records 4 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? there is no pass mark; the number is only meaningful beside the outcome it did or did not predict, and on this kit it predicts it backwards

Cadence: whenever a station is re-run

The decisionWhen to reach for it

In one linePropagation

of the item verdicts that came out wrong, how many had already lost their evidence in the purchased transcript

MethodHow the test was run

evals/scoring.py::words_survived and ::propagation. For each wrong item the gold SPAN — the exact turn text and speaker that item rests on — is looked for in the purchased text with a windowed WER threshold of 0.34, and separately the item is asked whether the rule that decided it needed a speaker the transcript never labelled. The denominator is WRONG items, never all items, and the rate is UNDEFINED when nothing was wrong.

Every grader on these pages scored the same 64 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

three readings, one answer. The interesting part is the prohibited item: every content word of the claim is in the transcript, and the rule that a refused claim was not asserted is what keeps all three arms from calling it a breach. A keyword check flags it.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl spans — the turn text and the speaker each item verdict rests on, written by the corpus builder when the call was generated.

No true/false rates for this grader. It records 2 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? 0.6 and 0.2 are the bands evals/scoring.py::propagation uses to word its own reading; they are editorial, not pass marks. The 0.34 window is the span test's threshold and it is deliberately generous, which is what makes the result a lower bound

Cadence: on every scored run

The decisionWhen to reach for it