The business caseThe problem this solves

Someone has a body of written material — product documentation, policies, contracts, runbooks — and a stream of questions whose answers are already in there somewhere. Today those questions are answered by a person who knows where to look, or by nobody. Reading forty documents to find the paragraph that answers one question — the kit retrieves the passages and drafts the answer with them quoted beside it, so the check is reading five passages instead of forty files.

The inputThe actual documents

The corpus is 40 documents, 1.45 MB (docx 3 · html 12 · md 12 · pdf 5 · txt 8). Five formats deliberately, so extraction is exercised rather than assumed. It is SQLite's own public documentation, chosen because it is genuinely public domain and can be redistributed inside a public repository.

The outcomeWhat a good result looks like

A reader asks a question in their own words. The system finds the five passages most likely to contain the answer, shows them, and drafts an answer that cites which passage it used. The evidence and the answer arrive together, so the check is reading five passages rather than forty files.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• A release gate — a wrong answer must not ship — Exact substring
  It almost never waves a failure through: TNR 1.000 and 0.833. Over 100 adjudicated answers 14 of its 15 mistakes are false FAILURES, which waste review time rather than shipping a regression — but 1 wrong answer did get past it, so it is a strong gate and not an infallible one.
• Cheap triage over high volume, where a false alarm is expensive — Token F1
  TPR 1.000 on both models and free, so it almost never flags a good answer. Use it to shrink what a human or a judge has to read.
• The criterion needs interpretation — tone, faithfulness, whether a paraphrase counts — LLM judge, zero-shot
  The only grader here that reads meaning. It is what caught the refusal the substring check scored correct.
• You need to know whether it is hallucinating — Faithfulness
  It asks a different question from correctness — does the answer follow from the passages — so it catches invention that a correctness grader scores as right.
• The data cannot leave your network — Any code grader
  All four code graders run in-process, cost nothing and send nothing. Three of them beat the null baseline.
• Ordered or procedural answers, where sequence carries meaning — ROUGE-L
  It is the only code grader that rewards ORDER, via longest common subsequence.
• The judge disagrees with you and you cannot tell who is right — Human review
  It is the only grader that is actually right, and the other six are calibrated against it. Adjudicating only the DISAGREEMENTS is the cheap way in — 15 rows here rather than 100 — but it cannot give either grader a rate, because the sample is chosen on the very thing being measured. On those 15 the substring check scores 0%; over all 100 it scores 85.0%.

At a glanceHow the whole thing runs

Run twice over the same set, for real, the last on 2026-08-01. Every figure on these pages was captured from those runs — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 1 question: clone (a fresh clone of this kit runs with nothing fetched).

Last verified 2026-08-01 — two models, one provider, one key — re-judged 2026-08-01. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Verified from a cold clone on the day of the run: 40 documents, 668 chunks, and the committed retrieval numbers reproduced exactly, with no key configured.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. end-to-end per answered row: retrieval plus the model call, nearest-rank over the 50 answered rows, the same convention evals/run.py uses for its model-only figures

Current processWhat it replaces

Reading forty documents to find the paragraph that answers one question — the kit retrieves the passages and drafts the answer with them quoted beside it, so the check is reading five passages instead of forty files.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• html
• md
• txt
• pdf
• docx
• 1 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one chunk is

paragraph-greedy to a hard 1800-character ceiling, after corpus-wide boilerplate removal

The indexWhat the index build measured

There IS an index here, and this is the one kit of the fifteen that builds one: src/index.py writes 668 chunks to chunks.json in 0.36s, and it is a file on your disk rather than a service. Retrieval is stdlib TF-IDF over that file, top-5, with no key and no network — which is why the build costs $0.00. It is deterministic, so a rebuild that changes the index is a real diff rather than noise.

LicenceLicence

Public domain. SQLite's dedication names documentation as well as code, verified against sqlite.org/copyright.html on 2026-07-31. No attribution clause, no share-alike, no notice file.

Bring your ownBring your own documents

Replace the files in data/corpus/ and run python -m src.index. That is the whole procedure: the index is a file, it is deterministic, and the rebuild takes under a second for forty documents. The labelled set in data/labelled.jsonl is about THESE documents, so your accuracy numbers start meaningless until you write your own rows — evals/check_labels.py enforces six rules on them and will tell you what is wrong before you spend anything on a run.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

Token budgetWhere the tokens actually go

The five retrieved passages are 1,761 of 1,911 input tokens; the reader’s own question is 21. That ratio is the cost lesson, and it is why a per-query bill is driven by how much context you retrieve.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 1,911 tokens assembled, 1,761 are passages — 92% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Full promptThe prompt in full

You answer questions using only the provided document passages.
If the passages do not contain the answer, say so plainly and do not guess.
Cite the passage number you used, like [2].

Answer the question using only these passages.

[1] whentouse (pdf)
Appropriate Uses For SQLite
Table Of Contents
SQLite is not directly comparable to client/server SQL database engines such as MySQL, Oracle, PostgreSQL,
or SQL Server since SQLite is trying to solve a different problem. Client/server SQL database engines strive to implement a shared repository of enterprise data. They emphasize
scalability, concurrency, centralization, and control. SQLite strives to provide local data storage for individual
applications and devices. SQLite emphasizes economy, efficiency, reliability, independence, and simplicity. SQLite does not compete with client/server databases. SQLite competes with fopen(). 1. Situations Where SQLite Works Well
Embedded devices and the internet of things
Because an SQLite database requires no administration, it works well in devices that must operate without
expert human support. SQLite is a good fit for use in cellphones, set-top boxes, televisions, game consoles,
cameras, watches, kitchen appliances, thermostats, automobiles, machine tools, airplanes, remote sensors,
drones, medical devices, and robots: the "internet of things". Client/server database engines are designed to live inside a lovingly-attended datacenter at the core of the
network. SQLite works there too, but SQLite also thrives at the edge of the network, fending for itself
while providing fast and reliable data services to applications that would otherwise have dodgy
connectivity.

[2] quirks (docx)
o has stumbled over some quirk of SQLite that is not mentioned here, please let the developers know by posting a brief message on the SQLite Forum. 2.

SQLite Is Embedded, Not Client-Server
Whenever comparing SQLite to other SQL database engines like SQL Server, PostgreSQL, MySQL, or Oracle, it is important first of all to realize that SQLite is not intended as a replacement or competitor to any of those systems. SQLite is serverless. There is no separate server process that manages the database. An application interacts with the database engine using function calls, not by sending messages to a separate process or thread. The fact that SQLite is embedded and serverless instead of being client/server is a feature, not a bug. Client/server databases like MySQL, PostgreSQL, SQL Server, Oracle, and others are an important component of modern systems. These systems solve an important problem. But SQLite solves a different problem. Both SQLite and client/server databases have their role. Developers who are comparing SQLite against other SQL database engines need to clearly understand this distinction. See the Appropriate Uses For SQLite document for additional information. 3. Flexible Typing
SQLite is flexible with regard to datatypes. Datatypes are advisory rather than mandatory. Some commentators say that SQLite is "weakly typed" and that other SQL databases are "strongly typed". We consider these terms to be inaccurate and even pejorative. We prefer to say that SQLite is "flexibly typed" and that other SQL database engines are "rigidly typed". See the Datatypes in SQLite document for a detailed discussion of the type system in SQLite. The key point is that SQLite is very forgiving of the type of data that you put into the database.

[3] faq (docx)
n many NFS implementations. You should avoid putting SQLite database files on NFS if multiple processes might try to access the file at the same time.

On Windows, Microsoft's documentation says that locking may not work under FAT filesystems if you are not running the Share.exe daemon. People who have a lot of experience with Windows tell me that file locking of network files is very buggy and is not dependable. If what they say is true, sharing an SQLite database between two or more Windows machines might cause unexpected problems. We are aware of no other embedded SQL database engine that supports as much concurrency as SQLite. SQLite allows multiple processes to have the database file open at once, and for multiple processes to read the database at once. When any process wants to write, it must lock the entire database file for the duration of its update. But that normally only takes a few milliseconds. Other processes just wait on the writer to finish then continue about their business. Other embedded SQL database engines typically only allow a single process to connect to the database at once. However, client/server database engines (such as PostgreSQL, MySQL, or Oracle) usually support a higher level of concurrency and allow multiple processes to be writing to the same database at the same time. This is possible in a client/server database because there is always a single well-controlled server process available to coordinate access. If your application has a need for a lot of concurrency, then you should consider using a client/server database. But experience suggests that most applications need much less concurrency than their designers imagine. When SQLite tries to access a file that is locked by another process, the default behavior is to return SQLITE_BUSY.

[4] whentouse (pdf)
exibility since
new columns and indices can be added without having to recode every query. Stand-in for an enterprise database during demos or testing

Client applications typically use a generic database interface that allows connections to various SQL
database engines. It makes good sense to include SQLite in the mix of supported databases and to
statically link the SQLite engine in with the client. That way the client program can be used standalone
with an SQLite data file for testing or for demonstrations. Education and Training
Because it is simple to setup and use (installation is trivial: just copy the sqlite3 or sqlite3.exe executable
to the target machine and run it) SQLite makes a good database engine for use in teaching SQL. Students
can easily create as many databases as they like and can email databases to the instructor for comments or
grading. For more advanced students who are interested in studying how an RDBMS is implemented, the
modular and well-commented and documented SQLite code can serve as a good basis. Experimental SQL language extensions
The simple, modular design of SQLite makes it a good platform for prototyping new, experimental
database language features or ideas. 2. Situations Where A Client/Server RDBMS May
Work Better
Client/Server Applications
If there are many client programs sending SQL to the same database over a network, then use a
client/server database engine instead of SQLite. SQLite will work over a network filesystem, but because
of the latency associated with most network filesystems, performance will not be great. Also, file locking
logic is buggy in many network filesystem implementations (on both Unix and Windows).

[5] whentouse (pdf)
ms) to analyze large datasets. Raw data can be imported from CSV files, then that data can be
sliced and diced to generate a myriad of summary reports.

More complex analysis can be done using
simple scripts written in Tcl or Python (both of which come with SQLite built-in) or in R or other
languages using readily available adaptors. Possible uses include website log analysis, sports statistics
analysis, compilation of programming metrics, and analysis of experimental results. Many bioinformatics
researchers use SQLite in this way. The same thing can be done with an enterprise client/server database, of course. The advantage of SQLite
is that it is easier to install and use and the resulting database is a single file that can be written to a USB
memory stick or emailed to a colleague. Cache for enterprise data
Many applications use SQLite as a cache of relevant content from an enterprise RDBMS. This reduces
latency, since most queries now occur against the local cache and avoid a network round-trip. It also
reduces the load on the network and on the central database server. And in many cases, it means that the
client-side application can continue operating during network outages. Server-side database
Systems designers report success using SQLite as a data store on server applications running in the
datacenter, or in other words, using SQLite as the underlying storage engine for an application-specific
database server. With this pattern, the overall system is still client/server: clients send requests to the server and get back
replies over the network. But instead of sending generic SQL and getting back raw table content, the client
requests and server responses are high-level and application-specific.

Question: Why is SQLite not directly comparable to client/server engines such as MySQL, Oracle or PostgreSQL?

Raw responseThe raw response

SQLite is not directly comparable to client/server engines because it is trying to solve a different problem: SQLite provides local data storage for individual applications and devices, while client/server databases emphasize scalability, concurrency, centralization, and control [1]. Also, SQLite is embedded and serverless, not a client/server system [2].

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Answer customer questions from your product manuals — 50 questions drawn from 40 real documents. Two tiers of one model family answered, and every answer was then graded Seven different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

A further 91 questions were later authored and run as a probe — an attempt to build a harder set that could separate the graders better. They are not part of any score on this page, and what they found is under The specification.

The rulerHow it was graded

A MODEL DECIDES, and this is the only kit here where that is true — every other one grades with code. The judge's verdict is an opinion, so unlike an exact match it can be wrong in both directions, and the rates on this page move when it is. That is why seven rulers were run over the same 100 answers rather than one: the choice of ruler is itself a decision, and putting a grader that passes everything on the same table is what makes the others readable. The judge is the reference standard the other six are scored against, so nothing on this page can check the judge — only the adjudication can, and it was another model.

MeasurementsWhat was measured

The adjudication behind the substring figure — all 100 answers, the confusion matrix and the rows the graders fought over — is on Judge validation.

Three words carry this page: Counted is deterministic and nobody's opinion, Judged is a model's verdict that moves if the grader is wrong, and Not yet known is printed blank rather than filled with something plausible. The row most worth having is currently the empty one.

Run it twiceThe same set, run again

Two runs, and the ordering reversed. Same fifty questions, the same index, the same judge model and the same prompt: the fast tier finished ahead on the first run and behind on the second, by exactly one row each way. Neither ordering is published as a result, because neither is one. Two tiers a single row apart on fifty questions are not distinguishable on this set, and it took a second run to show it.

judged correct (LLM-as-judge) — The two runs are not averaged into one figure. An average of 90.0 and 88.0 is 89.0, which reads as a measurement nobody took and hides the only thing this pair actually established — that the gap is smaller than the noise. The runs are printed as runs.

Grading costWhat it costs

Every dollar published on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate. It is a projection onto a rate card, not a bill anyone paid. Token counts belong to this pipeline — top_k, chunk size, prompt — and hold wherever you run it; the dollars belong to whoever you buy from, which is why the card is named beside every figure.

Rates checked 2026-08-01. A cheaper American model than Gemini 2.5 Flash-Lite almost certainly exists — Amazon Nova Micro is widely reported at $0.035 and $0.14 per million. AWS's own pricing table would not render to an automated fetch on three attempts, so it is named here and left unpriced rather than quoted from a comparison site.

Grading unitWhat the grading figure prices

GRADING COSTS MONEY HERE, AND ON EVERY OTHER KIT IT IS FREE. The unit is tokens because the grader is itself a model call: 337 in and 246 out per row on the fast tier, over 54 calls for 50 rows, since an unreadable verdict escalates to a higher ceiling. So the promise made elsewhere on this site — that a fresh clone can re-score the committed records for $0.00 — does NOT hold for the judge on this kit. What is free here is re-running the six code graders over the recorded answers; re-running the judge is a fresh bill.

The gradersSeven ways to grade

The judge passes about 90% of these rows, so a grader that returns “correct” to everything already agrees with it 90% of the time. That is the number every grader has to beat, and two of them do not.

The centre line is a grader that passes everything, not zero. Four beat it. Two tie it by construction. One scores below it — the exact-substring check that this kit originally graded correctness with.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Set limitationsWhat this set cannot show

Of 50 rows, 43 are judged correct on BOTH models. Those carry no signal about grader quality — every grader passes them, so they inflate agreement and separate nothing. Three rows are correct on exactly one model. Only FOUR (L26, L31, L41, L46) are wrong on both, and those four are the entire discriminating power of the set.

This is why every winning threshold landed at the bottom of its sweep, and why TNR moves by 0.17-0.20 when a single row flips. The set is not too small — it is too EASY, and adding more of the same rows would not help.

The attempt above cost $0.025 for 91 rows on one tier plus judging, against the $0.06 estimated for 50 rows on two. The balanced set still does not exist, and this is now a measured gap rather than an untried plan: authoring harder questions is not the lever. A negative class big enough to alarm on needs a harder TASK -- a corpus with genuine ambiguity, or questions spanning two documents -- not a longer list of questions about this one.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Technique choiceHow the grader was built

An LLM judge is one evaluation technique. The other five — deterministic, human review, weighted rubric, error analysis and hybrid — and the dashboard each one needs are on Evals, and a worked judge walkthrough is on the Evals dashboard.

Correctness here is judged by a model, and that is a decision with a history. The original grader marked an answer correct if it contained a required fragment. That is cheap, deterministic and needs no key — and it produced four different wrong numbers, each discovered only after fixing the last: table-of-contents fragments, then fragment length, then terseness, then paraphrase rigidity.

The tell was not the score moving. It was the ranking inverting. Noise moves a number; an artifact reorders the comparison the page exists to make. At that point the honest move is to change the instrument, not to relabel a fifth time.

Three things keep the replacement honest, and all three are checkable:

• The judge never sees the required fragment.It gets the question, a reference answer and the candidate. Showing it the fragment would rebuild the instrument being replaced.
• It was proved RED before it was trusted green.A self-test sends four fluent, confident wrong answers — including one that is true but answers a different question — plus two correct paraphrases that must pass. Six of six behaved as required. A grader that cannot fail anything reports 100% and means nothing.
• The substring check still runsrenamed to what it actually measures: grounded. It fails in the opposite direction — it cannot be talked round by a confident wrong answer, and it cannot recognise a paraphrase. Every disagreement between the two is recorded.

One more thing worth stating, because it argues against the judge: a token ceiling covers thinking, not just output. A 400-token cap on a one-sentence verdict lost rows to empty and truncated replies, and re-rolling at the same ceiling reproduced the failure exactly. The ceiling now escalates, and only an unreadable verdict is retried — retrying a “false” would be shopping for the answer you want.

The full run log ships in the kit repo at results/judge-run-2026-08-01.log: the self-test block, every disagreement with the judge’s reason, and the row where the substring grader scored a refusal as correct because the model quoted the passage while declining.

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — top_k, chunk size, prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-01. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same question, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingGrading is a second bill on the same traffic

GRADING IS A SECOND BILL ON THE SAME TRAFFIC, AND IT IS ABOUT 39 PERCENT OF THE COMBINED ONE. The judge reads a question, a reference answer and a candidate — roughly 337 input tokens against the 1,492 the pipeline sends — so priced on a single card it costs LESS per row than answering does, not more. It still costs a second call, and its ceiling escalates on an unreadable verdict, so it made 54 and 56 calls for 50 rows rather than 50. A forker re-running the eval pays it ON TOP of the pipeline, and a code grader makes the column zero. Note what changed here: this page used to say grading cost MORE than answering, which was true only because the judge ran on the dearer of two tiers. On one rate card that reverses.

Cost driversWhat actually moves the bill

• RETRIEVED CONTEXT, and it is not close. On the worked example the five passages are 1,761 of 1,911 input tokens; the user's question is 21. Doubling top_k very nearly doubles the bill, and the reader's question length barely moves it.
• top_k — 5 here. It is one constant in src/retrieve.py and it is the largest cost lever in the kit.
• The chunk ceiling (1800 chars). Bigger chunks mean fewer, larger passages; the product of k and chunk size is what you actually pay for.
• Output length, which is the only cost difference between the two tiers once both are priced on the SAME rate card. The deliberating tier writes about twice as much — 298 tokens against 155 — for a judged score that is within one row of the fast tier's and that swapped places with it between two runs. Bigger gaps than that between two models are usually the rate card talking, not the workload.
• Output length. The deliberating tier wrote ~298 tokens per answer to the fast tier's ~155, and output is priced at 2x input on both.
• Grading, if you re-run the eval — see cost_of_evaluation_usd. It is the line people forget and it is larger than the pipeline itself.

Your volumeWhat it costs at your volume

Linear. Nothing here batches and nothing amortises: ten times the questions is ten times the bill, because each query retrieves different passages and so shares no cacheable prefix beyond the 121 tokens of system prompt and chat framing. The index is built once and costs nothing to serve, so corpus growth raises latency and memory before it raises the bill.

Model choiceWhy this model

This kit runs on the cheapest capable model available, deliberately. The whole programme is self-funded community work, and every figure on this page was produced for about eleven cents — which is the point: the pipeline, the labelled set, the error taxonomy and the grading discipline are what transfer, and none of them depend on paying frontier prices to learn. What a run costs YOU depends on the model you already hold a key for, so the projection below prices the same measured workload across the models you are more likely to be running in production.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on this page: 50 labelled questions answered by two models, and all 100 of those answers graded by the LLM-as-judge.

The codeEvery file, and what it is for

10 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/extract.pyextract — a swap seam

PDF/DOCX/HTML/MD/TXT to text; pypdf is the one dependency in the kit

You change it to: add a format. The corpus is deliberately five formats so this seam is exercised rather than asserted.

# SEAM 2 — text out of a document. Add a format by adding one entry to EXTRACTORS.

class HtmlText(HTMLParser):
def from_html(path, md=False):
def from_plain(path):
def from_docx(path):
def from_pdf(path):
EXTRACTORS = {
def extract(path):

src/boilerplate.pyboilerplate

drops furniture repeated ACROSS the corpus — 909 lines over 36 distinct patterns

# Drop the navigation, headers and footers that every document in a corpus repeats.

SHARE = 0.5
MAX_LEN = 160
MIN_DOCS = 3
SMALL = 5
def _repeated(texts):
def detect(docs, formats=None):
def strip(docs, formats=None):

src/chunk.pychunk — a swap seam

splits to a hard 1800-char ceiling; 668 chunks, p50 865

You change it to: MAX is a hard ceiling, read by the UI rather than retyped. All five recorded failures live behind this seam.

# SEAM 3 — how a document becomes retrievable pieces.

TARGET = 900
OVERLAP = 150
MIN = 120
MAX = 1800
def _cap(part, ceiling=None):
def split(text):
def chunk_document(doc_id, text, fmt):

src/index.pyindex

writes chunks.json; deterministic, so a rebuild that changes the index is a real diff

# Build and load the index. A file, not a server.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
INDEX = os.path.join(HERE, "data", "index")
CHUNKS = os.path.join(INDEX, "chunks.json")
VECTORS = os.path.join(INDEX, "vectors.json")
NOT_CORPUS = {"SOURCES.md", "README.md"}
def build(corpus_dir=CORPUS, verbose=True):
def load():

src/retrieve.pyretrieve — a swap seam

stdlib TF-IDF, top-5, no key and no network

You change it to: keyword and embedding sit behind one call; the index records which built it, and dispatch is on what the index HAS, not on what .env asks for.

# SEAM 4 — which passages the model gets to see.

TOP_K = 5
def tokenize(s):
def build_keyword_index(chunks):
def keyword(question, chunks, idf, k=TOP_K):
def cosine(a, b):
def embedding(question, chunks, vectors, embed_fn, k=TOP_K):
def retrieve(question, index, cfg=None, k=TOP_K):

src/prompt.pyprompt — a swap seam

assembles the prompt AND its decomposition in one pass, so they cannot disagree

You change it to: change the wording and the parts change with it — the decomposition is built as the string is built.

# SEAM 5 — what the model actually receives.

SYSTEM = (
def assemble(question, passages):

src/adapters/__init__.pyadapters — a swap seam

one completion call over raw HTTP; every provider returns the same shape including usage

You change it to: PROVIDER, BASE_URL, MODEL in .env. Two adapters ship: openai-compatible, which covers every provider speaking the OpenAI chat-completions shape plus any local server (Ollama, LM Studio, vLLM), and anthropic, which is its own shape. Adding one is a function and a dict entry.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
def complete(cfg, system, user, max_tokens=1024):

src/app.pyapp

stdlib http.server UI; holds no pipeline logic

# The UI server. One command, no dependency, no build step, and it runs with no API key.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
RESULTS = os.path.join(HERE, "results")
LABELS = os.path.join(HERE, "data", "labelled.jsonl")
PORT = int(os.environ.get("PORT", "8765"))
STATE = {}
def _norm(s):
def load_state():
def api_status():

evals/run.pyrun

the eval harness: retrieval pass, answer pass, cause taxonomy

# The eval harness. Scores the labelled set and classifies every failure.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
def _present(fragments, text):
def classify(row, hits, index, docs):
def retrieval_pass(rows, index, docs, k):
def answer_pass(rows, index, results, cfg, k, limit=None):
def summarise(results, k):
def main():

evals/judge.pyjudge

LLM-as-judge for correctness, with a self-test that must go red first

# The correctness grader. Reads a completed run and re-scores every answer with a model.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
VERDICT_TOKEN_SCALE = (400, 1200, 3000)
SYSTEM = (
def _read_verdict(text):
def _verdict(cfg, question, reference, candidate):
SELF_TEST = [
def self_test(cfg):
def main():

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 1492 input and 155 output tokens per question at top-k 5, of which about 150 tokens are fixed (system prompt, chat framing, the question itself) and the rest is retrieved context. Change the retrieval depth and the input scales with it. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model and top-k follow the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

A local, single-user tool over public documents. Binds 127.0.0.1, no auth, no rate limit, no session — correct for a demo on your own machine and wrong for anything else.

Read from the environment only, never written into the repo, never requested from a reader on any surface.

The experimentWe attacked it — two gates

An indirect prompt injection has to clear two gates, and they behave differently. Both were run for real on 2026-08-01, keyword retriever, top_k 5, 41-document scratch corpus.

The naive payload lost the retrieval race outright. Rewritten to repeat the query’s own terms — the oldest trick there is — the same instruction took rank 1.

The resultHijack failed, poisoning worked

Neither model ever replied with the canary. But 5 of 10 answers repeated “defragment” — a claim that exists only in the hostile document and nowhere in SQLite's real documentation — and cited it as a passage.

HonestyWhat this does not prove

• One provider, one day, one payload, ten samples. A refusal rate is not a defence.
• No test of a payload written to survive a semantic retriever rather than a keyword one.
• Nothing was tested against a corpus containing private or personal data.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

You answer questions using only the provided document passages.
If the passages do not contain the answer, say so plainly and do not guess.
Cite the passage number you used, like [2].

src/prompt.py — it is the system prompt, and it is the only guardrail in the kit.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• A prompt rule is a request, not enforcement — nothing rejects an answer that ignores it.
• No output filter, no schema validation on the answer, no PII redaction.
• No defence against a hostile passage: the injection run put an instruction into the prompt verbatim and 5 of 10 answers took content from it.
• No rate limit and no authentication in front of any of it.

WatchedWhat is watched, and why that one

The kit measures 60 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 36 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

12 measured by the latest run 48 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (rows 50, top_k 5) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

5 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked.

DeviationsWhat deviated

2 breaches across 5 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Grade a sample on every release, and the whole set when the model, the prompt, the chunker or top_k changes — those four are the seams that move the number. A full re-run costs about $0.21 per 1,000 rows on the published Flash-Lite card.

In one lineNo framework, deliberately

This kit has no framework and one dependency, pypdf. That was a choice, and the point of it is that you can read the prompt.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing in this pipeline loops, branches or retries: one pass, seven seams. A graph earns its place when a cycle appears — re-retrieve on a bad answer, route by question type, call a tool. On this flow it would be a straight line with extra vocabulary.

The other sideWhat a framework costs you

• A dependency tree, and a version treadmill attached to it.
• An abstraction between you and the prompt, on a page whose whole argument is that you should see the prompt.
• Defaults you did not choose — a chunk size, a template, a retry policy.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-pro on the deliberating tier, 2026-08-01. This kit records telemetry measured per run, split by stage — 6 of the 6 readings on this axis, 6 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 4 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Where the time goesRetrieval against the model

The only stage split anything in this estate records. It is two stages, not a span tree — the pipeline is not instrumented per step, and nothing here should be read as though it were.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

1 run not plotted. r002-retrieval recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: a deterministic chunk index in a file on your disk.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, JUDGE_MODEL, MAX_CALLS_PER_DAY, MODEL, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: Public domain. SQLite's dedication names documentation as well as code, verified against sqlite.org/copyright.html on 2026-07-31. No attribution clause, no share-alike, no notice file. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineExact substring

Does the answer contain a required fragment, verbatim?

MethodHow the test was run

evals/run.py, in-process, no key. The fragment comes from answer_contains in data/labelled.jsonl.

Every grader on these pages scored the same 100 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The answer is right. It says “run at the same time” where the label says “proceed concurrently”, so the substring grader fails it and every other grader passes it.

The formulaWhat it computes

fragment.lower() in answer.lower()

IterationsFour rulers, four wrong numbers

The tell was the ranking inverting, not the value moving. Noise moves a number; an artifact reorders the comparison the page exists to make.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: adjudicated truth, not another grader — all 100 answers were read and given a verdict, including the 85 the two graders already agreed on, so these are accuracy and not agreement. The adjudicator is itself a model, so this is model-confirmed and not human-confirmed.

How tight can the band be? Over all 100 answers this check is right 85 times, and its error is almost entirely in one direction: 14 answers it failed that the adjudication scored correct, against 1 it passed that was wrong. The negative class is 11 rows of 100 — wider than the 5 and 6 a single model gives, but one row still moves TNR by 0.09. The band stays one row wide and the denominator is printed beside every rate.

Cadence: Grade a sample on every release, and the whole set when the model, the prompt, the chunker or top_k changes — those four are the seams that move the number.

The decisionWhen to reach for it

In one lineToken F1

How much of the reference wording overlaps the answer, in both directions?

MethodHow the test was run

Offline over the committed result files. Content words only; stopwords dropped.

Every grader on these pages scored the same 100 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The answer is right. It says “run at the same time” where the label says “proceed concurrently”, so the substring grader fails it and every other grader passes it.

The formulaWhat it computes

harmonic mean of precision and recall over the multiset of content-word tokens

A continuous score needs a decision threshold to become pass/fail, and the threshold is a knob someone picks. This one was swept 0.05 to 0.95 in steps of 0.05 and the best-agreeing value was ≥ 0.05. The sweep is published with the result, because a single tuned number shown alone is the same trick as quoting a test score you tuned on.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the LLM judge, zero-shot — so these are agreement split by DIRECTION, not accuracy against truth.

How tight can the band be? The reference has only 5 and 6 negatives on 50 rows, so ONE row flipping moves TNR by 0.20 and 0.17. Any alarm band finer than one row is alarming on noise — which is why the band here is one row wide and the denominator is printed beside it.

Cadence: Grade a sample on every release, and the whole set when the model, the prompt, the chunker or top_k changes — those four are the seams that move the number.

The decisionWhen to reach for it

In one lineReference recall

How much of the reference answer's content made it into the answer?

MethodHow the test was run

Offline over the committed result files.

Every grader on these pages scored the same 100 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The answer is right. It says “run at the same time” where the label says “proceed concurrently”, so the substring grader fails it and every other grader passes it.

The formulaWhat it computes

share of the reference's content words that appear in the answer

A continuous score needs a decision threshold to become pass/fail, and the threshold is a knob someone picks. This one was swept 0.05 to 0.95 in steps of 0.05 and the best-agreeing value was ≥ 0.05. The sweep is published with the result, because a single tuned number shown alone is the same trick as quoting a test score you tuned on.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the LLM judge, zero-shot — so these are agreement split by DIRECTION, not accuracy against truth.

How tight can the band be? The reference has only 5 and 6 negatives on 50 rows, so ONE row flipping moves TNR by 0.20 and 0.17. Any alarm band finer than one row is alarming on noise — which is why the band here is one row wide and the denominator is printed beside it.

Cadence: Grade a sample on every release, and the whole set when the model, the prompt, the chunker or top_k changes — those four are the seams that move the number.

The decisionWhen to reach for it

In one lineROUGE-L

Does the answer follow the reference's wording in the same ORDER?

MethodHow the test was run

Offline over the committed result files.

Every grader on these pages scored the same 100 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The answer is right. It says “run at the same time” where the label says “proceed concurrently”, so the substring grader fails it and every other grader passes it.

The formulaWhat it computes

F-measure over the longest common subsequence of content words

A continuous score needs a decision threshold to become pass/fail, and the threshold is a knob someone picks. This one was swept 0.05 to 0.95 in steps of 0.05 and the best-agreeing value was ≥ 0.05. The sweep is published with the result, because a single tuned number shown alone is the same trick as quoting a test score you tuned on.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the LLM judge, zero-shot — so these are agreement split by DIRECTION, not accuracy against truth.

How tight can the band be? The reference has only 5 and 6 negatives on 50 rows, so ONE row flipping moves TNR by 0.20 and 0.17. Any alarm band finer than one row is alarming on noise — which is why the band here is one row wide and the denominator is printed beside it.

Cadence: Grade a sample on every release, and the whole set when the model, the prompt, the chunker or top_k changes — those four are the seams that move the number.

The decisionWhen to reach for it

In one lineLLM judge, zero-shot

Is the answer correct, judged on meaning rather than wording?

MethodHow the test was run

evals/judge.py against a completed run. The judge NEVER sees the fragment — showing it would rebuild the instrument being replaced.

Every grader on these pages scored the same 100 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The answer is right. It says “run at the same time” where the label says “proceed concurrently”, so the substring grader fails it and every other grader passes it.

The promptWhat the judge is asked

6 of 6 — four fluent wrong answers rejected, two correct paraphrases accepted

The analysisWhat it actually did

54 calls for 50 rows and 56 for 50; the extra calls are ceiling escalations, because a token cap covers thinking and not just output

In operationWhat to monitor

Reference standard: none, because this grader IS the reference standard. Every other correctness grader on these pages is scored against its verdicts, which is exactly why its own cannot be scored from anything in this table.

How tight can the band be? The reference has only 5 and 6 negatives on 50 rows, so ONE row flipping moves TNR by 0.20 and 0.17. Any alarm band finer than one row is alarming on noise — which is why the band here is one row wide and the denominator is printed beside it.

Cadence: Grade a sample on every release, and the whole set when the model, the prompt, the chunker or top_k changes — those four are the seams that move the number.

The decisionWhen to reach for it

In one lineLLM judge, few-shot

The same question, with two worked examples in the prompt.

MethodHow the test was run

Same 100 recorded answers, same judge model, two examples added: a terse polar answer that must pass, and a fluent wrong one that must fail.

Every grader on these pages scored the same 100 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The answer is right. It says “run at the same time” where the label says “proceed concurrently”, so the substring grader fails it and every other grader passes it.

The promptWhat the judge is asked

The zero-shot criterion, unchanged, with two worked examples appended: a terse polar answer that must pass, and a fluent wrong one that must fail.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the LLM judge, zero-shot — so these are agreement split by DIRECTION, not accuracy against truth.

How tight can the band be? The reference has only 5 and 6 negatives on 50 rows, so ONE row flipping moves TNR by 0.20 and 0.17. Any alarm band finer than one row is alarming on noise — which is why the band here is one row wide and the denominator is printed beside it.

Cadence: Grade a sample on every release, and the whole set when the model, the prompt, the chunker or top_k changes — those four are the seams that move the number.

The decisionWhen to reach for it

In one lineFaithfulness

Does the answer follow from the passages it was given — regardless of whether it is correct?

MethodHow the test was run

The retrieved passages were RE-DERIVED (the result files record chunk ids, not text) and the derivation proved: regenerated chunk ids matched the recorded ones on 50 of 50 rows, both models, before anything was judged.

Every grader on these pages scored the same 100 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The answer is right. It says “run at the same time” where the label says “proceed concurrently”, so the substring grader fails it and every other grader passes it.

The promptWhat the judge is asked

4 of 4 — a fabricated number, a flat contradiction and a claim that is TRUE but absent from the passages were all rejected; a genuine paraphrase passed

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: none in this table, and that is not an omission. Every other grader here answers "is the answer correct" and is scored against the zero-shot judge. This one asks whether the answer follows from its passages, so agreeing or disagreeing with a correctness judge would mean nothing in either direction. It was checked by a 4-case red-proof instead.

How tight can the band be? It flagged nothing at all on the fast tier and one answer on the deliberating tier. A negative class with no members cannot produce a rate, and one with a single member swings the whole way if that row flips — so there is no alarm band here worth drawing, and the counts are printed instead of a percentage. The 4-case red-proof is the evidence that it can say no; 100 percent is not.

Cadence: Grade a sample on every release, and the whole set when the model, the prompt, the chunker or top_k changes — those four are the seams that move the number.

The decisionWhen to reach for it

Read this firstWhat this measures, and what it does not

Error analysis reads the system’s failures. Adjudication settles a fight between two graders. This is the third thing — using those settled rows to score the graders. The system is not on trial here; the rulers are.

Each row was read twice by a second model — once plainly, once by a reader told to hunt for a reason to fail it. The two passes agreed on 15 of 15. That is a useful signal and it is not human confirmation. These numbers have since been written onto the Evals pages: the substring check now publishes 85.0% against adjudicated truth, and the judge’s own rates stay UNKNOWN, because a grader cannot be scored against itself and the adjudicator here is another model. ingest_adjudication.py was not the path and is superseded — its whole scope is the 15 disputed rows.

ScoresHow each grader scored

Every bar carries its denominator. Read the substring row against the earlier version of this page: on the 15 disputed rows it scored 0 of 15 and looked worthless. Over all fifty it is 85% accurate. It is not a bad grader — it is a grader that loses every argument it picks, which is a different and much more useful thing to know.

The four outcomesCounts, before any rate

Read the second column first. It is the expensive one: a grader that says correct about a wrong answer publishes a number and nothing looks red. A grader that says wrong about a correct answer only wastes a review. The substring check has the only entry in that column, and it is row 11.

Not yet knownWhy the judge scores 100%

The substring check does not have this problem: it is deterministic and shares nothing with either. Its 85% is the more trustworthy figure on this page. The one row that would settle the judge’s score is a person reading a sample — which is why that card stays dashed and ingest_adjudication.py is still unrun.

If you grade another waySix methods, and what each one needs

This kit used an LLM judge. That is one method, not the method. If you grade by hand, by rubric, by panel or by a deterministic script, your inputs, your outputs, the way you validate the method, and the dashboard you need are all different. Open any row.

Cost is denominated in the method’s own unit. A human reviewer costs minutes, not tokens — a manual method reported as $0.00 is not cheap, it is unpriced.

They are kept on a separate page on purpose. Those panels are templates — illustrative shapes for methods we did not run — and a template sitting beside a measurement is how the two get mistaken for each other.

The casesThe fifteen disputed rows, in full

Click a case for the answer given, the reference, the fragment the substring check wanted, and both readings. Nothing here is summarised away.