The business caseThe problem this solves

Every organisation has a database and a queue of people who cannot query it. The questions are not hard — how many, which region, what was the total last quarter — but they need SQL, so they need a person who writes SQL, and that person becomes a bottleneck for questions that take them ninety seconds each. A ticket to whoever writes SQL. The question is asked in a message, joins a queue, and comes back a day later as a number with no query attached — so the next person to ask the same question asks again.

The inputThe actual database rows

The corpus is 5400 database rows, 0.37 MB (sqlite 1). ⚑ ON OPERATOR DIRECTION, THE SITE'S OWN TABLES. The alternative was a public benchmark database, which would have brought a licence to read, a download to host and a schema nobody here can plant a trap in. Generating it means the traps are DELIBERATE and known: 112 customers with no 2024 orders (so an inner join silently drops them), 135 orders with a NULL amount (so SUM and COUNT disagree), and exactly one shared display name (so grouping by name is not grouping by customer). A corpus that cannot express the failure cannot show the eval layer earning its keep.

The outcomeWhat a good result looks like

The question is answered in about 2.7 seconds, and the SQL comes back with the rows — so the answer can be checked, corrected and re-used rather than trusted.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Letting non-technical people ask questions of a small, well-named schema — the fast tier
  80.0% at 2.7 seconds and $0.000140 a question, against a 10.0% floor. The two tables here are exactly the shape most internal databases start as.
• Anything where the answer moves money or goes in front of a customer — the fast tier, with the query shown and a person reading it
  The query is the audit trail. A wrong number with its SQL beside it is correctable in seconds; the same number alone is not.
• Choosing between the two tiers this kit actually ran — the fast tier, and read the query
  80.0% against 75.0% on the identical questions, and it is the cheaper of the two to run at your own rates once you count its tokens. The dearer tier's one real win is that it never returned an empty reply.

And where nothing here is good enough:

• A warehouse with hundreds of tables — not this kit, yet
  The whole schema goes in the prompt. That is a design choice which is honest at two tables and stops working long before a hundred.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-11. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-11 — r010-data-qa-flash + r011-data-qa-pro. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — python3 tools/build_db.py then python3 -m src.app. No pip install, no network, no key — the database is generated locally and the recorded run is committed, so a fresh clone renders real output before it has spent anything.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. the model call only — one call per question, 20 questions run serially. The DATABASE is not in this number and it is not a rounding error that it is absent: executing the generated query took a median of 0.50 ms and never more than 2.12 ms across the whole run. Better than 99% of the wait is the model writing the query, which is why the only latency lever on this kit is the model and not the index, the hardware or the schema.

Current processWhat it replaces

A ticket to whoever writes SQL. The question is asked in a message, joins a queue, and comes back a day later as a number with no query attached — so the next person to ask the same question asks again.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusCorpusCorpusEvalsEvalsPromptPromptPromptPromptArchitectureArchitectureArchitectureArchitectureApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• db
• 2 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one question is

none — one question is one unit and goes whole into one call

⚠︎ THESE ARE GUARD VALUES AND THE FIELD NAMES ARE RETRIEVAL'S. There is nothing to chunk and nothing to retrieve: the sizes are the assembled PROMPT in characters, which barely varies because the schema card is fixed and only the question changes. They are recorded because the guard's job is to decide whether two runs may be differenced, and a run against a rebuilt database is not the same measurement. There is no train/test split: all 20 labelled questions are scored, every time, because 20 is small enough that holding any of them back would measure noise.

SetupWhat the setup figure measured

There is no index. 0.05s is tools/build_db.py generating the whole database from its seed — and the rebuild is byte-identical, verified by comparing the file before and after. The schema card is read out of the database at call time, so there is no build step between the two.

LicenceLicence

MIT, with the rest of the kit. ⚑ A SELF-AUTHORED CORPUS REMOVES THE LICENCE ARGUMENT ENTIRELY — there is no third-party terms page to read, no attribution to carry and no share-alike to reason about, because nothing here came from anybody else. The generator is committed, so the corpus is reproducible rather than merely permitted.

Bring your ownBring your own database rows

Point src/execute.py's DB at your own SQLite file. The schema card is read out of whatever is there, so the prompt re-shapes itself with no code change. What you must supply is the labelled set: your questions and their gold SQL, in data/labelled.jsonl. Without it the kit still answers, and there is no number.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

Token budgetWhere the tokens actually go

Input is 49% of the tokens on an average question, and 94.5% of that input is identical on every call: the rules (419 chars) and the schema card (339 chars) between them, with the reader's own question at 5.5%. That is the opposite shape to a RAG kit, where the retrieved passages move with what you asked — and it means the lever here is the prompt and the schema, never the traffic. ⚠︎ The split is apportioned by CHARACTER SHARE at 3.02 chars per token; the provider returns one total per call, so the parts are derived rather than measured and the assembly table says so.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 823 characters assembled, 482 are instructions — 59% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Assembled by src/prompt.py::assemble at capture time from the committed schema card and q-01's question, which is the same code path the run used. The assembly checks that every part it publishes occurs in the prompt it sends, in order, so the decomposition below cannot drift from the text above.

Full promptThe prompt in full

You write SQLite queries. Given a database schema and a question, reply with ONE SQL statement and nothing else.
Rules:
- Reply with SQL only. No explanation, no markdown fences, no commentary.
- Use only the tables and columns in the schema.
- The query must be read-only: a single SELECT (or WITH ... SELECT). Never modify data.
- If the question cannot be answered from this schema, reply with exactly: CANNOT ANSWER

---

Database schema:

TABLE customers (
  customer_id INTEGER PRIMARY KEY,
  name TEXT NOT NULL,
  region TEXT NOT NULL,
  signup_date TEXT NOT NULL
)
TABLE orders (
  order_id INTEGER PRIMARY KEY,
  customer_id INTEGER NOT NULL,
  order_date TEXT NOT NULL,
  order_amount REAL,
  status TEXT NOT NULL
)
  FOREIGN KEY orders.customer_id -> customers.customer_id

Question: How many customers do we have?
SQL:

Raw responseThe raw response

SELECT COUNT(*) FROM customers;

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Answer a shop's sales questions from its order records — 20 questions drawn from 5400 real database rows. Two tiers of one model family answered, and every answer was then graded one way — by == against a fixed gold value, which is a measurement too and only as good as the gold behind it.

The rulerHow it was graded

EXECUTION MATCH. There is no single correct query — >= '2024-01-01' and strftime('%Y', d) = '2024' are the same answer written twice — so comparing SQL strings would score correct answers wrong. Both queries are RUN and their result sets compared. Column NAMES are ignored (AS revenue vs AS total_revenue is a labelling choice); column SETS are compared exactly, and that strictness is this ruler's known cost.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill.

Rates checked 2026-08-11. The provider that actually ran r010 is not priced here. It is not a card this site names on a page, and quoting a rate for it would put a number next to a vendor without a published source behind it.

Grading unitWhat the grading figure prices

Grading calls nothing: two queries are run against a local SQLite file and their rows compared. Re-scoring the recorded run is therefore free and repeatable — which is exactly what evals/rescore.py did when the ruler was corrected. ⚠︎ THE GRADING IS FREE; THE RUN IS NOT. Unlike a retrieval kit there is no offline half: producing a query to score requires the model.

The gradersOne way to grade, and why it is the only one

The floor is the first row of the table, not a footnote: 80.0% against 10.0% is the comparison, and 10.0% of it was free.

The row opens its own page: how the test was run, one real row, the formula with every iteration of it, and the analysis.

Choosing oneWhen to reach for the model at all

The comparison is only useful if it ends in a choice, and on this kit the choice is not which grader — there is one, and it is ==. It is whether to call a model for this field at all, and the answer is conditional.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-11. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same question, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingGrading is free here, and it is a measured zero

Scoring runs two SQL statements against a local file and compares the rows. Nothing is sent. That is what made the correction on this kit possible at zero cost: the recorded statements were re-judged under a fixed ruler without calling anything. ⚠︎ The GRADING is free; the RUN cost $0.002792 in projected tokens.

Cost driversWhat actually moves the bill

• The schema card — 339 characters on EVERY call, whatever the question was. It is 49% of an average question's tokens, it is the fixed cost of putting the whole schema in the prompt, and it grows with your DATABASE rather than with your traffic.
• The generated query — 281 output tokens on average, and the average is misleading. The median answer is short; two replies spent the entire 1024-token ceiling and returned nothing at all, and those two alone are 36% of the whole run's output tokens.
• The database — a measured ZERO, which is the useful part. Executing the query took a median of 0.50 ms against a local file: no index to build, no embeddings to pay for, no vector store to host. Every cent of this kit is the model.

Your volumeWhat it costs at your volume

Linear and boring, which is the useful finding. 10,000 questions cost about $1.40 on the Google Gemini 2.5 Flash-Lite card. There is no index to rebuild, no store that grows and no per-document ingest, so 10× the questions is 10× the bill and nothing else moves. 10× the DATABASE, however, moves the input side of every single call — which is the asymmetry worth planning around.

Model choiceWhy this model

One model, one key, run once — the rule everywhere in this series. The fast tier was chosen because the question here is whether a CHEAP model can write correct SQL against a small schema, not whether an expensive one can. It answered 80% correctly and returned nothing at all twice, and both of those are more useful than a higher number from a tier nobody would run at volume.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

Every number on these pages: 20 questions, one call each, on the fast tier. Two of the 20 replies returned no statement while spending the full output ceiling, and their tokens are in these totals — a projection that quietly dropped them would under-price the failure mode this kit found.

The codeEvery file, and what it is for

11 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

tools/build_db.pyThe database builder

Generates data/shop.db from a fixed seed — 400 customers, 5,000 orders — with three traps planted on purpose. No network, and byte-identical on every rebuild, which is what lets a reader reproduce a number rather than take one.

# Build the sample database. Two tables, one fixed seed, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DB = os.path.join(HERE, "data", "shop.db")
SEED = 20260811
N_CUSTOMERS = 400
N_ORDERS = 5000
NO_ORDER_CUSTOMERS = 112
NULL_AMOUNT_SHARE = 0.03
FIRST = ["Brightwell", "Kesler", "Ardan", "Nornis", "Vantage", "Halloway", "Pemberton",
LAST = ["Foods", "& Vance", "Logistics", "Supply", "Partners", "Holdings", "Metals",

src/schema.pyThe schema card

Reads the tables, columns, types and foreign keys OUT OF the database with PRAGMA and renders the 339-character card the prompt carries. It is read rather than written down, so it cannot drift from the database it describes.

# SEAM 2 — the schema card. The only thing about your database the model ever sees.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DB = os.path.join(HERE, "data", "shop.db")
def read(path=DB):
def card(tables=None, path=DB):
def stats(path=DB):

src/prompt.pyThe prompt

One assembly, published verbatim, with a checked invariant that every part it claims occurs in the prompt actually sent. Also owns CANNOT ANSWER — the refusal token, deliberately not SQL, so 'no answer exists' can never be recorded as 'wrote something dangerous'.

# SEAM 5 — what the model actually receives.

SYSTEM = (
CANNOT = "CANNOT ANSWER"
def assemble(question, schema_card):
def clean(text):

src/guard.pyThe read-only guard

THE NODE THAT MATTERS. An allowlist on the leading keyword — SELECT, or WITH resolving to a SELECT — plus a hard cap of one statement, with comments and string literals stripped before either is decided. Anything not positively recognised as a read is refused.

# SEAM 3 — the read-only guard. Pure code, and the reason this kit is safe to point at your data.

ALLOWED_LEAD = ("select", "with")
NAMED = ("insert", "update", "delete", "drop", "alter", "create", "replace", "truncate",
class Refused(Exception):
def strip_noise(sql):
def statements(sql):
def enforce(sql):
def check(sql):

src/execute.pyThe executor

Opens the database with SQLite's own mode=ro URI — the second independent layer, and the one that does not depend on anybody's parser being right — and runs the statement under a 5-second timeout and a 500-row cap, recording when either fires rather than truncating silently.

# SEAM 4 — running the statement. Pure code, read-only, bounded in time and in rows.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DB = os.path.join(HERE, "data", "shop.db")
TIMEOUT_S = 5.0
MAX_ROWS = 500
class ExecError(Exception):
def connect(path=DB):
def run(sql, path=DB, timeout_s=TIMEOUT_S, max_rows=MAX_ROWS):
def signature(result):
def same(a, b, ordered=False):

src/adapters/__init__.pyThe model adapter — a swap seam

Two providers behind one call, and the only place the shared daily call budget is checked and recorded.

You change it to: Two providers ship behind one function; a third is one more. What it costs you: re-run the labelled set. A score from one model says nothing about another, and this page would be quoting a number for a system that no longer exists.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
def complete(cfg, system, user, max_tokens=1024):

src/app.pyThe local UI

Binds 127.0.0.1. Renders the recorded run with no key at all, so the demo is honest about when it is not calling anything.

# The minimal local app — one of the four layers a kit is allowed.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
RESULTS = os.path.join(HERE, "results")
LABELS = os.path.join(HERE, "data", "labelled.jsonl")
PORT = int(os.environ.get("PORT", "8010"))
def api_status():
def api_connect(body):
def api_examples():
def api_ask(question):

evals/run.pyThe scorer

Execution match: the model's query and the gold query are both RUN and their result sets compared, because >= '2024-01-01' and strftime('%Y', d) = '2024' are the same answer written twice. Classifies every failure into one of seven causes, each with a different owner.

# The eval harness. Scores by EXECUTION MATCH and classifies every failure.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
def score_row(row, sql_text, card):
def summarise(results):
def main():

evals/baseline.pyThe floors

Three constants — always-refuse, count-star, select-all — scored by the same score_row the model's answers go through. Free, and the only number on this kit that can be reproduced without spending.

# What you get without a model. Run this BEFORE reading any score this kit produces.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
STRATEGIES = (
def score(constant, labels, card):
def main():

evals/redteam.pyThe red team

25 statements fired straight at the guard, no model involved, plus a live mode that measures whether a QUESTION can induce one. The offline half is free because the control it attacks is code.

# Attack the thing that protects the database, and say what it does and does not stop.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
ATTACKS = [
def verdict(row):
def offline():
def live(run_id, dry_run):
def main():

evals/rescore.pyThe re-scorer

Re-judges the recorded statements under a corrected ruler and writes a second file beside the first. Calls nothing. It exists because the model's output is evidence and the score is an opinion about it, and an opinion may be corrected without re-running anything.

# Re-score a recorded run against the current ruler. Calls nothing. Costs nothing.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
def rescore(path, note):
def main():

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 273 input and 281 output tokens per question, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

A local, single-user tool over a database you generated yourself. It binds 127.0.0.1, has no auth, no rate limit and no session — correct for a demo on your own machine and wrong for anything else. The only text a stranger controls is the QUESTION, and the only thing the question can reach is a model that writes SQL into a guard.

Read from the shared .env or the real environment only, never written into the repo, never requested from a reader on any surface. The repo has never held a credential. ⚠︎ THE KIT INHERITS THE SHARED .env AT THE REPO ROOT, so a fresh clone of THIS kit with no .env of its own can still have a live key — check /api/status before touching an endpoint that calls a model.

The experimentWe attacked the guard, not the prompt — because on this kit the guard is what stands between a question and your data

Every other kit in this series protects itself with a prompt, so measuring it means paying for generations. Here the control is src/guard.py: pure code, on the path between the model and the database. So the attack set skips the model entirely and fires the statements a compromised model could have written STRAIGHT AT THE GUARD — which is a stronger test of the guard than any number of natural-language attempts, because it assumes the attacker already won the argument with the model. It is also free, which is why it runs on every change rather than once. Measured on 2026-08-11, run redteam-offline: 25 statements fired straight at src/guard.py and src/execute.py with NO model involved, across 5 families including 7 harmless controls. Free to reproduce in about five seconds: python3 -m evals.redteam.

⚑ READ THE LAST FOUR ROWS TWICE. They are not failures and they are not passes — they are the guard's boundary, published rather than hidden. A legal SELECT may dump the whole customer table, and it may read the database's own DDL out of sqlite_master. The guard stops writes; it has nothing to say about which rows a read may touch. Whoever points this at real data needs that sentence more than they need the 14 refusals above it.

The result25 of 25 held — every write refused, every smuggling attempt refused, and no harmless statement wrongly refused. The two bounds fired where they should: a cross join was interrupted at 5000 ms and a 5,000-row read came back capped at 500 and marked truncated. ⚠︎ AND THE MODEL SIDE IS UNMEASURED: this says what happens IF a write is written, never whether a question can provoke one.

25 statements across 5 families. The 14 attacks that must be refused were all refused: four plain writes, six smuggling attempts (a second statement after a legal SELECT, a write hidden behind a line comment and behind a block comment, a CTE resolving to a DELETE, a case-mixed keyword, leading whitespace), and four attempts at the engine's own doors (ATTACH, PRAGMA, an open transaction, and the empty response). The 7 controls were all allowed, including the two that look dangerous and are not — a semicolon inside a string literal and the word DELETE inside a LIKE pattern. Those two are the reason strip_noise exists, and removing it turns one of them into a false refusal, measured.

HonestyWhat this does not prove

• ⚑ THE MODEL SIDE. Whether a natural-language question can talk this model into emitting a write is NOT measured. evals/redteam.py --live is written and costs one call per attack; it has not been run. Unmeasured is not safe, and this is the first thing to measure next.
• Whether the attack set is representative. 25 statements were authored, not sampled; an allowlist can only be shown to refuse the shapes somebody thought of.
• Whether the guard holds on other SQL dialects. It is written for SQLite's grammar, and WITH ... DELETE is legal in engines this kit never runs.
• Whether mode=ro holds under a SQLite build with unusual compile-time options. It is SQLite's own flag and is trusted here rather than independently verified.
• Anything about a deployed, multi-user version. There isn't one, and every number here was taken against a loopback server with no auth.
• Whether a legal read is an acceptable one. The guard permits SELECT * FROM customers; whether the person asking should see those rows is a database-permissions question this kit does not answer.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

One read-only SELECT, or nothing runs. The model is asked a question and returns a STATEMENT, and a statement can delete the table it was asked about — so the decision about whether it may touch the database is never the model's.

src/guard.py — enforce(), an allowlist on the leading keyword plus a hard cap of one statement, with comments and string literals stripped before either is decided. Called by src/app.py and evals/run.py on the same path, before src/execute.py.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• ⚑ IT IS NOT A LIMIT ON WHAT A LEGAL QUERY MAY READ, AND THIS IS THE SENTENCE TO CARRY AWAY. The allowlist is on SHAPE. SELECT * FROM customers is a perfectly good read and the guard allows it — measured: it returned all 400 rows in 0.36 ms. If the question 'show me every customer' should not be answerable by whoever is typing, the control for that is a database user with column and row permissions, and this kit does not ship one.
• It is not a schema-confidentiality control either. SELECT sql FROM sqlite_master is a read, so it is allowed, and it returns the DDL of tables the schema card never mentioned — measured at 400 rows on this database.
• It is not a syntax checker, deliberately. A parenthesised SELECT is allowed by the guard and then rejected by SQLite, and that division of labour is the design: a guard that also validated syntax would start refusing valid queries to protect a database from an error that cannot hurt it.
• It is not a defence against a compromised database file, a stolen key, or anyone with shell access to the machine. It binds 127.0.0.1 and has no auth of any kind.
• ⚠︎ IT IS NOT EVIDENCE THAT THE MODEL WILL NOT TRY. The offline red team assumes the statement was already written and asks what happens next. Whether a QUESTION can induce this model to emit a write is a separate measurement that has not been run.

WatchedWhat is watched, and why that one

The kit measures 23 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

12 measured by the latest run 11 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (answered 20, rows 20, schema_card_chars 339) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

2 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked.

not a time series No two of these 2 runs measured the same system — they differ on rescored_from, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

DeviationsWhat deviated

0 breaches across 2 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-run python3 -m evals.redteam on any change to src/guard.py or src/execute.py — it is free and takes about five seconds. Re-run the full labelled set on any change to src/prompt.py, the model, or the database, because all three change what is being measured rather than how.

In one lineNo framework, deliberately

This kit has no framework and no dependencies at all — sqlite3, urllib and http.server are standard library, and pip install is not a step in the instructions. That is a POSITION rather than an omission, and on a text-to-SQL kit it is a stronger one than usual: the frameworks in this space bundle exactly the two things that must be auditable here, the prompt and the guard, and both fit on one screen.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph, and the honest reason is that the pipeline has one path with one decision on it. schema → prompt → model → guard → execute, and the guard's only branch is 'or nothing runs'. There is no retry, no re-plan on a failed query, no self-correction loop — and those are precisely what a graph framework is for. ⚠︎ IF YOU ADD THE OBVIOUS NEXT FEATURE — feeding a SQLite error back to the model for a second attempt — that IS a cycle with a termination condition, and the argument for writing it by hand gets weaker. It is not implemented here, so the kit does not claim to have compared the two.

The other sideWhat a framework costs you

• A dependency tree, and a version treadmill on somebody else's release schedule.
• An indirection between you and the exact text sent to the model, on a kit whose report page publishes that text verbatim.
• Inherited defaults on the one control that protects the database. A framework flag you did not write is a guard you cannot red-team line by line.
• Install friction on a kit whose whole claim is git clone and run.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r011-data-qa-pro on the deliberating tier, 2026-08-11. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 1 earlier run on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — guard values wearing retrieval's field names; the database schema is the ground truth.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, MAX_CALLS_PER_DAY, MODEL, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT, with the rest of the kit. ⚑ A SELF-AUTHORED CORPUS REMOVES THE LICENCE ARGUMENT ENTIRELY — there is no third-party terms page to read, no attribution to carry and no share-alike to reason about, because nothing here came from anybody else. The generator is committed, so the corpus is reproducible rather than merely permitted. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineExecution match against the gold query's own result set

Do the rows the model's query returns equal the rows the gold query returns, compared as sets unless the question declares itself ordered?

MethodHow the test was run

evals/run.py::score_row, in-process, no key and no model. Runs both statements through the same guard and the same executor the app uses.

The inputOne real row, seen by every grader

The trivial case, shown because it is the one where every stage is visible at once: prompt assembled, one statement returned, guard allowed it, 1 row back in 0.16 ms.

The formulaWhat it computes

correct / answered, where correct means the model's query and the gold query return the same rows — as SETS unless the question declares itself ordered.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: this grader, against gold SQL hand-authored for a database whose generator is committed — never against another model's output. Anyone who clones the repo can re-execute every gold statement and see the rows it returns, which is a weaker guarantee than a published benchmark's labels and a stronger one than a judge nobody can inspect.

How tight can the band be? There is no alerting threshold here because there is no alerting: the kit is a local tool with one run on record. The numbers above are what you would watch if you ran it continuously, named honestly as a register rather than drawn as a dashboard nobody is reading.

Cadence: Re-run the whole labelled set on any change to src/prompt.py, the model, or the database — all three change what is being measured. Re-run evals/redteam.py on any change to src/guard.py or src/execute.py; it is free and takes five seconds.

The decisionWhen to reach for it