The business caseThe problem this solves

You wired a model to your own tools and it works, mostly. What you cannot tell from watching it is how often it reaches for a tool it does not need — latency and money for nothing — and how often it stops one step short and hands back half an answer that reads like a whole one. Nothing in the loop is deciding whether another call is worth making, and that decision is the entire cost model. The keyword routing rule that decides which backend a request goes to, written once and never re-measured against what it gets wrong.

The inputThe actual requests

The corpus is 120 requests, 0.00 MB (jsonl 1 · sqlite 1 · txt 6). A tool-choice corpus has to declare, for every request, the sequence that answers it — and that label only exists relative to THIS set of tools. There is no public benchmark of requests labelled against your four local tools, because the tools are the variable. Generating both together is the only way the labels can be true, and a self-authored corpus removes the licence argument entirely.

{"decline": false, "id": "q001", "text": "How many orders were placed in February?", "trap": "plain", "truth": ["shop_sql"]}
{"decline": false, "id": "q002", "text": "How many customers are on file?", "trap": "plain", "truth": ["shop_sql"]}
{"decline": false, "id": "q003", "text": "Which customer has spent the most overall?", "trap": "plain", "truth": ["shop_sql"]}
{"decline": false, "id": "q004", "text": "What does the returns policy say about the deadline?", "trap": "plain", "truth": ["doc_search"]}
{"decline": false, "id": "q005", "text": "How long is the warranty on electrical items?", "trap": "plain", "truth": ["doc_search"]}
{"decline": false, "id": "q006", "text": "When is shipping free?", "trap": "plain", "truth": ["doc_search"]}
{"decline": false, "id": "q007", "text": "What is 165 percent of 2600?", "trap": "plain", "truth": ["calc"]}
{"decline": false, "id": "q008", "text": "What is 150 multiplied by 4200?", "trap": "plain", "truth": ["calc"]}
{"decline": false, "id": "q009", "text": "What is today's date?", "trap": "plain", "truth": ["today"]}
{"decline": false, "id": "q010", "text": "What is the date today?", "trap": "plain", "truth": ["today"]}
{"decline": false, "id": "q011", "text": "How many orders fall outside the deadline in the returns policy?", "trap": "two-step", "truth": ["doc_search", "shop_sql"]}
{"decline": false, "id": "q012", "text": "What percentage of customers have placed more than one order?", "trap": "two-step", "truth": ["shop_sql", "calc"]}
{"decline": false, "id": "q013", "text": "How much did customer 11 spend, as a percentage of the February total?", "trap": "two-step", "truth": ["shop_sql", "calc"]}

The outcomeWhat a good result looks like

A request is answered in about 2.9 seconds and 1.75 model calls, with the chosen tool sequence recorded in order and scored against the labelled one.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• One request, one obvious tool, vocabulary you control — the keyword router you already have
  It gets every plain request right for nothing — 40 of 40 here — and a model buys you latency and a bill for those.
• Requests that need two tools in order — the model
  The router gets 0 of 24 at every setting. The second tool needs what the first returned, and no keyword rule can express that.

And where nothing here is good enough:

• Requests nothing you have can answer — neither, yet
  The router runs a tool on all 16. The model declines on 9 of 16 and runs a query on 6 — far better and not safe, because a confident empty result is the failure that looks most like an answer.
• Tools that write, or that are slow or expensive — not this kit
  All four tools here are local, read-only and instant, which is exactly when calling one wrongly costs nothing. Nothing here measures the case where it costs something.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-13. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-13 — r015-tool-pick-flash. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — A fresh clone with no key configured rebuilds the corpus, checks the labels and scores the free floor over all 120 requests at four settings — the whole argument for the kit, for $0.00. It cannot re-run the model.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. END-TO-END per REQUEST, summed across every model call that request made — not per call. ⚠︎ THAT IS THE DIFFERENCE FROM EVERY OTHER KIT HERE: a request that needed two tools made three model calls and its latency is all three. The tools themselves are local, read-only and effectively instant (a SQLite read, a grep over six short notes, arithmetic), so this clock is the model's time and nothing else — which also means it measures nothing about tool choice when a tool is slow.

Current processWhat it replaces

The keyword routing rule that decides which backend a request goes to, written once and never re-measured against what it gets wrong.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureEvalsEvalsEvalsEvalsPromptPromptPromptPromptArchitectureArchitectureArchitectureArchitectureApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• jsonl
• sqlite
• txt
• 3 of 11 stations carry a recorded failure

scroll → 11 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one request is

none — every request is judged, there is no held-out split

SetupWhat the setup figure measured

There is no index. The database is queried directly and the six notes are grepped, so 'building' the corpus is generating it — 0.4s and $0.00, byte-identical every time.

LicenceLicence

MIT — this repository's own licence, verified against LICENSE at the repo root on 2026-08-13. Nothing is fetched and no third-party grant is involved.

Bring your ownBring your own requests

Replace the four functions in src/tools.py with your own and re-label the requests against them. ⚠︎ The labels are relative to the catalogue: a fifth tool makes every existing sequence a claim about a different question.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 1,384 characters assembled, 1,265 are instructions — 91% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

The whole prompt as src/prompt.py assembled it for the FIRST call of q005 on r015-tool-pick-flash — the system segment, then the request with an empty transcript. On later calls the second segment carries every tool call and result so far, verbatim. ⚠︎ THE SYSTEM SEGMENT IS ALSO THE SOURCE OF THE RUN'S ONE LABEL DEFECT: “call a tool only when you cannot answer without it” told the model not to call calc for arithmetic it could do, while the corpus labelled calc REQUIRED. The re-score fixed the label; the prompt is where a future run should fix the ambiguity.

Full promptThe prompt in full

You answer a request by calling tools, one at a time, and you stop as soon as you can answer.

The tools you may call, and there are no others:
- shop_sql(a single read-only SELECT over tables customers(id,name,joined) and orders(id,customer_id,placed,amount,shipped)) — runs it and returns up to 20 rows
- doc_search(a few keywords) — returns the matching policy notes, in full — there are six and they are short
- calc(an arithmetic expression using digits and + - * / ( ) . %) — evaluates it and returns the number
- today(nothing) — returns the current date as YYYY-MM-DD

Reply with EXACTLY ONE line, and nothing else:
  CALL <tool> | <argument>   to call a tool
  DONE | <answer>            when you can answer, including from what you already know
  NONE | <why>               when no tool here can answer it and you should not guess

Rules:
- Call a tool only when you cannot answer without it. Calling one you did not need is a failure, not caution.
- Some requests need two tools in order, where the second needs what the first returned.
- Some requests need no tool at all. Answer those with DONE straight away.
- Some requests cannot be answered by these tools. Answer those with NONE. Do not guess and do not run a query that cannot contain the answer.

REQUEST: How many orders exceed the limit set in the returns policy?

You have not called anything yet.

Your one line:

Raw responseThe raw response

CALL doc_search | returns policy deadline

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Answer a shop team's questions from its orders and policies — 120 requests. One model answered, and every answer was then graded one way — by == against a fixed gold value, which is a measurement too and only as good as the gold behind it.

The rulerHow it was graded

The grader is src/score.py and it is pure code: the sequence the model chose against the labelled sequence, compared IN ORDER, producing one of six outcomes. ⚑ Order is part of the answer — shop_sql → calc and calc → shop_sql are not the same run, because the arithmetic needs the number the query returns, and a set comparison would score the impossible ordering as correct. ⚑ THE SAME MODULE SCORES THE FREE FLOOR, which is what makes the comparison a comparison rather than two measurements taken with different instruments.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill. ⚠︎ AND ON THIS KIT THE OUTPUT SIDE IS THE UNSTABLE HALF: the average reply carried 462 output tokens for a one-word answer, almost all of it this model's own reasoning, so a card for a model that answers without reasoning would overstate the cost several times over.

Rates checked 2026-08-13. The provider that actually ran r014 is priced from build/facts/models.json, which is the site's own dated model table, and it is the only figure on these pages that is what was actually paid rather than a projection.

Grading unitWhat the grading figure prices

Grading calls nothing — two lists are compared. A measured $0.00, not an unpriced one.

The gradersOne way to grade, and why it is the only one

The free floor is a keyword router — one regex per tool, one shot — scored over the same 120 requests by the same module, 0 calls. 46.7% at its best setting. ⚑ RAISING ITS CALL BUDGET MAKES IT WORSE, not better: it trades 12 stopped-early for 24 kept-going and settles at 43.3%, and above two calls nothing moves at all because no request matches a third rule. Its ceiling is its rules, not its budget. ⚑ AND A CONTROL, BECAUSE THE FLOOR PASSING A TRAP IS THE SUSPICIOUS RESULT: it gets all 12 wrong-surface requests right, and swapping the order of two rules turns that into 12/12 WRONG. It passes that trap on rule order, not on reading. The floor was not re-tuned on the strength of that.

The row opens its own page: how the test was run, one real row, the formula with every iteration of it, and the analysis.

Choosing oneWhen to reach for the model at all

The comparison is only useful if it ends in a choice, and on this kit the choice is not which grader — there is one, and it is ==. It is whether to call a model for this field at all, and the answer is conditional.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-13. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingGrading is free here, and it is a measured zero

Two lists are compared by src/score.py. Nothing is called, so this is $0.00 rather than approximately nothing. ⚑ AND THE FREE FLOOR IS FREE ON THE SAME TERMS: four call budgets over all 120 requests, scored by the identical module the model run uses — 0 calls, $0.00. So is the re-score, and so was diagnosing all 26 unusable replies from the recorded wire.

Cost driversWhat actually moves the bill

• ⚑ THE CALL COUNT, WHICH THE MODEL CHOOSES. 1.75 model calls per request on average, 210 in total. No other kit here has this line.
• Reasoning tokens. Output averaged 242 tokens per request for replies that are one line long, and the 26 requests that failed averaged three times the reasoning of the ones that worked.
• The transcript. Every tool result so far is re-sent on the next call, so input grows within a request as well as across the corpus.

Your volumeWhat it costs at your volume

Linear in requests, and cheap: 120 requests cost 210 model calls, so ten times the traffic is about 2100 calls. ⚑ BUT THE MULTIPLIER IS NOT 1 AND IT IS NOT FIXED — it was 1.75 model calls per request on this corpus, and a corpus with more two-step requests would raise it with no other change. Every other kit here scales by a constant; this one scales by a number the model picks.

Model choiceWhy this model

The fast tier, because it is what the other kits in this series ran and the rows are meant to be comparable. ⚠︎ It is also a REASONING model, and on this kit that is not neutral: its thinking is billed inside completion tokens and it consumed the 512-token ceiling on 26 of 120 requests.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

Every model number on these pages: 120 requests, 210 model calls, on the fast tier.

The codeEvery file, and what it is for

Six modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Three of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/tools.pyTool catalogue — a swap seam

the four tools the model may call and nothing else, rendered into the prompt from the same list the guard checks — so the prompt and the guard cannot describe different tools

You change it to: add or remove a tool and the prompt, the guard and the refusal message all follow from the one list. ⚠︎ Every label in the corpus is relative to THIS catalogue, so changing it invalidates the labelled sequences.

# The four tools the model may call, and nothing else.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
DB = os.path.join(DATA, "shop.db")
NOTES = os.path.join(DATA, "notes")
TODAY = "2026-03-31"
CATALOGUE = [
NAMES = [t["name"] for t in CATALOGUE]
class ToolError(RuntimeError):
def shop_sql(arg):

src/loop.pyThe loop — a swap seam

ask, parse one line, call, ask again — and stop. The step cap is here, and it is what bounds the bill on a kit where the model chooses the bill

You change it to: MAX_STEPS bounds calls per request. It is the comparability guard: two runs at different caps were not asked the same question.

# The loop: ask, parse, call, ask again — and stop.

MAX_STEPS = 4
MAX_TOKENS = 512
def parse(text):
def run_one(cfg, request_text, max_steps=MAX_STEPS):

src/loop.pyParser — a swap seam

one line, three shapes (CALL / DONE / NONE). Anything else ends the request as no_verdict rather than being guessed at

You change it to: MAX_STEPS bounds calls per request. It is the comparability guard: two runs at different caps were not asked the same question.

# The loop: ask, parse, call, ask again — and stop.

MAX_STEPS = 4
MAX_TOKENS = 512
def parse(text):
def run_one(cfg, request_text, max_steps=MAX_STEPS):

src/router.pyKeyword router — a swap seam

the free floor: one regex per tool, one shot, scored by the same module as the model

You change it to: one regex per tool, and their ORDER — which is load-bearing, see the control in evals/baseline.py.

# The free floor: a keyword router, which is what most teams actually ship before a model.

RULES = [
DEFAULT_CAP = 1
def route(text, cap=DEFAULT_CAP):
def score(requests, cap=DEFAULT_CAP):
def sweep(requests, caps=(1, 2, 3, 4)):
TIERS = ("floor", "fast", "mid", "frontier")
COLD_N = 20
WILSON_Z = 1.959963984540054
PUBLISHABLE = ("Anthropic", "Google", "Meta", "OpenAI", "xAI")

src/score.pyScorer

the six outcomes, compared as SEQUENCES because order is part of the answer

# What counts as right, and the five ways a tool-choosing run goes wrong.

OUTCOMES = ("correct", "wrong_tool", "stopped_early", "kept_going", "should_have_declined",
MEANS = {
def outcome(req, got, replied=True):
def tally(rows):
def by_trap(rows):
def traps_handled(rows):

tools/build_corpus.pyCorpus builder

the seeded database, the six notes and the labelled requests, byte-identical on every rebuild

# Build the corpus: a seeded shop database, six policy notes, and 120 labelled requests.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
NOTES = os.path.join(DATA, "notes")
DB = os.path.join(DATA, "shop.db")
REQUESTS = os.path.join(DATA, "requests.jsonl")
SEED = 20260813
TODAY = "2026-03-31"
NOTE_TEXT = {
SQL, DOC, CALC, TODAY_T = "shop_sql", "doc_search", "calc", "today"

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 808 input and 242 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

A local, single-user tool over a corpus you generated yourself. It binds 127.0.0.1, has no auth because it has no second user, and reads a SQLite file from disk.

Read from the shared .env or the real environment only, never written into the repo, never logged, never returned by any endpoint.

The experimentWe attacked the tools, and named the surface we did not test

⚑ THE GUARDS WERE EXERCISED DIRECTLY, WITH NO MODEL INVOLVED: a DELETE, a stacked SELECT …; DROP TABLE, and __import__('os') through the calculator. All three refused, by the guard rather than by the model's good manners. ⚠︎ WHAT WAS NOT TESTED is a request crafted to talk the model into misusing a permitted tool — a query designed to exfiltrate rather than to answer. That surface is real and unmeasured, and it is smaller here only because nothing in the corpus is sensitive. Measured on 2026-08-13. The structural facts below are code-checked; no injection rate is claimed, because none was measured.

⚑ READ THE LAST ROW AGAINST THE OTHERS. The first three are structural and hold by construction. The fourth is a judgement the model makes 6 times out of 16, and it is the only one that can produce a confident wrong answer rather than a refusal.

The result⚑ THE ATTACK SURFACE HERE IS THE TOOL CATALOGUE, AND THE CONTAINMENT IS THAT NOTHING IN IT CAN ACT. Every other kit in this series is handed a document and asked about it; this one hands a model a set of capabilities and lets it choose. The whole safety argument is that all four are read-only, local, and bounded — a single SELECT with a row cap, a grep over six files, arithmetic on a character whitelist, and a constant.

Four call budgets over all 120 requests through the identical src/score.py the model run uses, plus a rule-order control. 0 calls, $0.00.

HonestyWhat this does not prove

• The prompt-injection rate. Not run, not authorised, and deliberately not estimated.
• Whether a tool result could carry an instruction the model then follows. Every result here comes from data this repo generated, so the question is real and untested.
• Whether the row cap is enough. A query returning 20 rows is carried into every later call in the request; nothing measures what a deliberately wide query costs.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

⚑ EVERY TOOL IS GUARDED BEFORE IT RUNS, AND THE CATALOGUE IS CLOSED. A model that names a tool outside the four gets a refusal back, not an exception and not a guess at what it meant. A query that is not a single read-only SELECT never reaches sqlite. The calculator sees only digits and operators. And the loop stops at 4 model calls per request whatever the model says, which is the only thing standing between a bad exit condition and an open-ended bill.

src/tools.py — call(), resolve() and the three tool guards; src/loop.py — MAX_STEPS and the parser.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is not a permission system. Every tool is read-only by construction, so there is nothing to authorise; the moment a tool can write, this whole design needs rethinking.
• It is not a content filter. Nothing inspects what the model says, only its shape.
• It is not a confidence score. NONE is a choice the model makes, not a threshold.

WatchedWhat is watched, and why that one

The kit measures 19 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

13 measured by the latest run 6 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

1 recorded run. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 13 chips that all say so.

DeviationsWhat deviated

0 breaches across 1 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-run python3 -m evals.check_labels on any change to src/tools.py — it exercises the tools against the labels rather than inspecting them — and python3 -m evals.baseline on any change to src/router.py. Both are free.

In one lineNo framework, deliberately

No framework and no dependencies at all — json, re, sqlite3 and urllib from the standard library, and requirements.txt is an empty file with a comment explaining that the emptiness is load-bearing. ⚠︎ THIS IS THE KIT MOST LIKELY TO BE ASKED TO CROSS THAT LINE, because an agent loop is exactly what agent frameworks sell.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

It IS a loop, and it is 40 lines: ask, parse one line, call, append, ask again, stop at the cap. That is the whole of what a framework would give you here, minus the scheduler, the callback registry and the tracing you would then have to configure. What a framework WOULD buy is what this kit deliberately does not have — retries with state, parallel tool calls, and tools that write.

The other sideWhat a framework costs you

• No dependency resolution, no lockfile, no framework upgrade path — and no framework features either. Anything this kit does not do, a forker writes.
• The loop is hand-written, so retries with state, parallel tool calls and streaming are absent rather than configured off.
• The reply format is a parsed line rather than a schema. ⚑ THAT COST THIS RUN 26 OF 120 REQUESTS: every unusable reply was an unparsed line, not a wrong choice, and structured output would have turned most of them into answers.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r015-tool-pick-flash on the fast tier, 2026-08-13. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. This is the only run on record, so there is nothing to move against. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — a closed catalogue of tools is the search space.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, MAX_CALLS_PER_DAY, MODEL, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT — this repository's own licence, verified against LICENSE at the repo root on 2026-08-13. Nothing is fetched and no third-party grant is involved. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe chosen tool sequence against the labelled one

Did the model call exactly the tools the request needs, in the order it needs them, and then stop?

MethodHow the test was run

src/score.py::outcome, in-process, no key and no model. The same function evals/baseline.py calls for the free floor and evals/rescore.py calls to re-score the recorded outputs.

The inputOne real row, seen by every grader

two steps in order: read the 30-day limit out of the notes, then count against it. The free router calls one of the two and stops, at every setting.

The formulaWhat it computes

sequence_exact = correct / (requests - no_verdict). ⚑ SEQUENCES, NEVER SETS — shop_sql → calc and calc → shop_sql are different answers and only one can work. ⚠︎ IT EXCLUDES no_verdict, which is why `answered` is published beside it: on THIS kit a model that returns nothing calls no tools, and calling no tools is the correct answer for every request that needed none, so a broken run scores points.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The corpus's own labels, written by tools/build_corpus.py when each request was generated and exercised — not merely inspected — by evals/check_labels.py, which RUNS the labelled tools against the shipped data. ⚑ THIS GRADER IS THE REFERENCE, so its own error rate is unknown by definition. ⚠︎ AND ON THIS RUN THE LABELS WERE WRONG ONCE: calc was labelled required while the prompt told the model not to call a tool it did not need. evals/rescore.py corrects that against the recorded outputs and both files are kept.

How tight can the band be? There is no threshold on the MODEL path: the sequence it called decides. The call budget belongs to the free floor and is swept over four settings rather than chosen — with a rule-order control proving its best trap result is luck.

Cadence: Re-run on any change to src/prompt.py, src/tools.py or the corpus generator. The free half (evals/check_labels.py, evals/baseline.py) costs nothing and should be re-run on every change to src/router.py.

The decisionWhen to reach for it