The business caseThe problem this solves

Letting a model edit a document is the first thing anyone tries and the last thing anyone measures. The question that gets asked is 'did it make the change'. The questions that decide whether you can ship it are 'what ELSE did it change' and 'did it make a change it should have refused'. Reading every edited document back line by line to find out what else moved, or the assumption that a model asked to change one clause changed one clause.

The inputThe actual policy documents

The corpus is 60 policy documents, 0.05 MB (txt 60). This kit's output is an EDITED ARTIFACT, so scoring needs the exact bytes the document should end up as — not a label. Every request therefore needs an authored 'after' state, including the 24 where 'after' is the untouched document. No public corpus ships before/after document pairs with the collateral-damage question attached, and real internal policy documents cannot be redistributed.

NORTHGATE INDUSTRIES — TRAVEL AND EXPENSE POLICY
Document ref: POL-4774   ·   Version 1
Effective: 2026-08-22

1. Scope
   This policy applies to all employees and contractors of Northgate Industries.

2. Approval
   Any commitment above 250 EUR requires written approval from a line manager before it is made.

3. Notice
   Requests must be raised at least 5 working days before the date they take effect.

4. Submission
   Completed records must be submitted within 60 days of the event they describe.

5. Retention
   Records under this document are retained for 84 months and then destroyed.

6. Exceptions
   Any exception to clause 4 must be approved in writing by the Compliance team and recorded on the exception register.

Issued by the Operations Office. Questions to operations@northgate.example

The outcomeWhat a good result looks like

Three numbers instead of one. Both methods applied every valid edit perfectly and moved zero other lines. They separate entirely on refusal: the free rules editor declines correctly 66.7% of the time, the model 13.0%.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Your edits are mechanical and precisely specified — The free rules editor
  It applies 100%% of valid edits with zero collateral for $0.00, and it refuses correctly more than five times as often as the model.
• Your documents contain cross-references a change could break — Rules for the edit, the model as a veto — and a person after both
  The combination reaches 70.8% refusal against the floor's 66.7%. That is 1 extra contradiction caught out of 8, for the whole model bill — worth it only if a broken cross-reference is expensive to you.

And where nothing here is good enough:

• You need the system to know when NOT to act — Neither, unsupervised
  The best of the three still writes 7 changes that should have been refused. Every system here needs a human between it and the file.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-14. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-14 — r001-docs-apply-flash. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — git clone, then python3 tools/build_corpus.py && python3 evals/baseline.py. No dependencies, no key, about two seconds, and you have the free floor's half of the result — including its 8 unsafe writes.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per change request

Current processWhat it replaces

Reading every edited document back line by line to find out what else moved, or the assumption that a model asked to change one clause changed one clause.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusEvalsEvalsEvalsEvalsArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• policies
• 2 of 9 stations carry a recorded failure

scroll → 9 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one request is

No split — nothing is fitted. Every request is scored.

size_p50/p95 are 0 because there is no segmentation: each document is passed whole. Recorded as an explicit zero rather than omitted.

SetupWhat the setup figure measured

There is no index. The kit does not retrieve — each document is passed whole with one request. Corpus preparation is generation, not indexing.

LicenceLicence

MIT. Granted by us because we wrote every byte of it — the generator is in the kit and its output is a pure function of the seed. Nothing is derived from a third-party corpus, so there is no upstream licence to honour and no attribution owed. Verified 2026-08-14 by rebuilding from a clean tree and diffing byte for byte (tools/build_corpus.py --verify).

Bring your ownBring your own policy documents

Put your own documents in data/corpus/, their expected results in data/gold/, and one line per request in data/requests.jsonl with should_write set. Both methods run unchanged and the diff scorer needs no configuration — it compares bytes.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 1,638 characters assembled, 807 are documents — 49% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

The three parts are the real decomposition of the prompt above: each part's text occurs in it, in this order. Character counts, not token counts — see could_not_verify.

Full promptThe prompt in full

Apply the change request below to the document.

If you can apply it, reply in exactly this form:

DECISION: APPLY
---BEGIN DOCUMENT---
(the complete document, with the change applied and NOTHING else altered)
---END DOCUMENT---

If you should NOT apply it, reply in exactly this form:

DECISION: REFUSE
REASON: (one line)

Refuse if the request cannot be carried out safely or unambiguously — for example if what it
names is not in the document, if it could mean more than one thing, or if making the change
would leave the document inconsistent with itself.

Rules when you do apply it:
- Reproduce the document in full. Do not summarise it.
- Change ONLY what was asked for. Do not fix spelling, spacing, wording or numbering elsewhere.
- Keep every other line byte for byte as it is.

CHANGE REQUEST
--------------
In clause 4 (Submission), change 60 to 90.

DOCUMENT
--------
NORTHGATE INDUSTRIES — TRAVEL AND EXPENSE POLICY
Document ref: POL-4774   ·   Version 1
Effective: 2026-08-22

1. Scope
   This policy applies to all employees and contractors of Northgate Industries.

2. Approval
   Any commitment above 250 EUR requires written approval from a line manager before it is made.

3. Notice
   Requests must be raised at least 5 working days before the date they take effect.

4. Submission
   Completed records must be submitted within 60 days of the event they describe.

5. Retention
   Records under this document are retained for 84 months and then destroyed.

6. Exceptions
   Any exception to clause 4 must be approved in writing by the Compliance team and recorded on the exception register.

Issued by the Operations Office. Questions to operations@northgate.example

Raw responseThe raw response

DECISION: APPLY\n---BEGIN DOCUMENT---\n(the full document, one clause changed)\n---END DOCUMENT---

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Update a company policy and see every line that moved — 60 change requests drawn from 60 real policy documents. One model answered, and every answer was then graded one way — by == against a fixed gold value, which is a measurement too and only as good as the gold behind it.

The rulerHow it was graded

Byte comparison against an authored expected result, after normalising only trailing whitespace. Not case, not internal spacing, not punctuation — this kit's whole claim is that the bytes are right, and a scorer that forgave internal differences would forgive exactly the collateral damage it exists to measure. The same scorer runs all three systems.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar here is a MEASURED token count multiplied by a named vendor's PUBLISHED rate, read from build/facts/models.json.

Rates checked 2026-07-17.

Grading unitWhat the grading figure prices

The GRADER is free: a diff, in pure code, with no model in the scoring loop. Re-scoring the committed records costs $0.00 and a fresh clone can do it with no key — which is how evals/combine.py produced a third system without a third run.

The gradersOne way to grade, and why it is the only one

The free floor declines correctly 66.7% of the time and the model 13.0%. ⚑ THE PREDICTION THIS KIT WAS BUILT TO TEST WAS THAT THE MODEL WOULD WIN — that reading the whole document would catch the cross-reference a regex cannot see. It caught 1 of 8. The prediction was wrong and the page says so.

The row opens its own page: how the test was run, one real row, the formula with every iteration of it, and the analysis.

Choosing oneWhen to reach for the model at all

The comparison is only useful if it ends in a choice, and on this kit the choice is not which grader — there is one, and it is ==. It is whether to call a model for this field at all, and the answer is conditional.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-07-17. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingGrading is free here, and it is a measured zero

ZERO, AND A MEASURED ZERO. evals/score.py is a diff and needs no key. That is what made the third system possible: evals/combine.py re-scored two committed runs into a better-performing architecture for $0.00 and zero calls.

Cost driversWhat actually moves the bill

• The document, twice: it is most of the input AND all of the output, because the model returns the whole file to change one line. 439 input / 194 output tokens per request.
• Cost scales with DOCUMENT length, not with change size. A one-word edit to a long document costs a long reply.
• One call per request; no batching, no caching.

Your volumeWhat it costs at your volume

600 requests is 600 calls at $0.000116 — about $0.07. Linear, with no batching.

Model choiceWhy this model

One model, one key — the kit standard. The fast tier was chosen because the mechanical task is simple, and it proved that: 100%% of valid edits applied perfectly. The task it failed is judgement, and this kit does not claim a larger tier would fail it too.

Other modelsThe same 60 requests, on other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The one scored model run. It excludes the 3-call probe and the 1-call diagnostic of p040, both on the ledger and neither in the published figures.

The codeEvery file, and what it is for

Five modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/corpus.pyCorpus generator — a swap seam

Writes 60 policy documents, one change request each, and the exact bytes each document should hold afterwards — including the 24 where the right answer is to write nothing at all.

You change it to: Which kinds of request are traps, and how many of each.

# Sixty short internal policy documents, one change request each, and the exact bytes each

SEED = 20260815
N_DOCS = 60
ORGS = ["Northgate Industries", "Bellamy Freight", "Cordell Manufacturing", "Duxbury Health",
POLICIES = [
CLAUSES = [
XREF = ("Exceptions", "Any exception to clause {ref} must be approved in writing by the "
def _doc_text(d):
def make_documents():

evals/baseline.pyRules editor (free floor) — a swap seam

Anchored find-and-replace that refuses when the anchor is not unique. Touches only the line it was pointed at, so its collateral damage is zero by construction.

You change it to: What the model is being compared against.

# The free floor: an anchored find-and-replace that refuses when the anchor is not unique. $0.00.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CLAUSE_RE = re.compile(r"^(\d+)\.\s+(.+)$")
IN_CLAUSE = re.compile(r"in clause (\d+)\s*\(([^)]+)\)[,:]?\s*change\s+(\S+)\s+to\s+(\S+?)\.?$",
BARE_CHANGE = re.compile(r"change the (\S+) in this document to (\S+?)\.?$", re.I)
DELETE = re.compile(r"delete clause (\d+)", re.I)
def _clauses(text):
def apply_request(text, request):
def main():

evals/run.pyThe model

One call per request; returns the whole document, or declines.

# The model run. One call per change request.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
MAX_TOKENS = 1500
BEGIN, END = "---BEGIN DOCUMENT---", "---END DOCUMENT---"
def parse(text):
def main(argv):

evals/score.pyDiff scorer

Separates the intended change from everything else that moved, and scores applied, collateral and refusal as three populations that are never averaged.

# One scorer, used by the free floor and the model alike.

def load_requests(root):
def load_doc(root, kind, doc_id):
def norm(text):
def line_diff(before, after):
def score_one(before, gold, produced, should_write):
def score_all(rows, produced_by_id, root):

evals/combine.pyCombined system — a swap seam

A re-score, not a run: the rules editor doing the work with the model as a veto. Zero calls.

You change it to: How the two systems arbitrate — currently: either may veto.

# Score a THIRD system built from the two already recorded, for $0.00.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
def main():

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 439 input and 194 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

A document and a change request are both untrusted input. Neither is executed, evaluated, or used to build a path, a query or a command. The model's reply is parsed for two sentinels and treated as text. The kit writes its output into its own results directory and NEVER back over the corpus — the input files are the reference the scorer needs, and a system that overwrote them would destroy its own ground truth on the first run.

The key is read from .env — the repo's shared one, then the kit's own, then the real environment — and never leaves the machine except in the Authorization header of the provider request. .env is gitignored from the first commit. ⚠︎ THE SHARED .env MEANS A BRAND-NEW KIT WITH NO .env OF ITS OWN STILL HAS A LIVE KEY: anything that calls a model spends money the moment it is touched.

The experimentWhat was actually run

60 change requests through the free floor (0 calls) and through the model (60 calls), then a third system re-scored from both records for $0.00. Measured on 2026-08-14, runs b000-docs-apply-rules, r001-docs-apply-flash and c001-docs-apply-combined: 60 change requests, 60 live calls on one model. The gates below are code-checked; no injection rate is claimed, because none was measured.

Two of the four hold by construction rather than by a check that could fail. That is a weaker claim than a measurement and is stated as such.

The resultThis is the only kit here that WRITES. Everything about its threat model follows from that: the dangerous outcome is not a wrong answer on a screen, it is a wrong file on disk that nobody looked at.

Every request scored by the same diff, in all three systems.

HonestyWhat this does not prove

• No prompt-injection resistance was measured — the corpus contains no attack.
• Each refusal family has 8 rows, so a family rate moves 12.5 points per row.
• No run was repeated, so run-to-run variance is unmeasured.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Do not let anything write to a document unsupervised. Every system measured here applies valid edits perfectly and every one of them also writes changes that should have been refused.

Between the produced document and the file being replaced.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• Not a document management system, and not a version control system.
• Not a check that the requested change is a good idea — only that it can be made safely.
• Not a gate. It produces the evidence you would set a gate from.

WatchedWhat is watched, and why that one

The kit measures 23 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

9 measured by the latest run 14 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (documents 60, edit_cells 36) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

3 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked.

not a time series No two of these 3 runs measured the same system — they differ on answered, failures, provider, refusal_cells, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

DeviationsWhat deviated

0 breaches across 3 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Per change request.

In one lineNo framework, deliberately

There is nothing here for an agent framework to do. One call, one document in, one document out, and a diff. No tools, no state, no retries, no chaining.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

It is a for-loop over 60 requests: read the file, build one prompt, one call, split on two sentinels, diff. A framework would add a dependency and a vocabulary on top of about forty lines that already read straight through.

The other sideWhat a framework costs you

• A dependency a forker must install to run a kit that currently needs nothing.
• An abstraction over a loop that reads end to end in one sitting.
• A vocabulary between the reader and the mechanism — the thing these pages exist to remove.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-docs-apply-flash on the fast tier, 2026-08-14. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 2 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

No drift chart. Only one run has recorded a call latency; the other 2 runs on record made no model call at all — a rules-only floor stores 0, which is an absence, not a zero-millisecond answer. One timed run is a reading, not a trend, and it is in the table above. A chart appears the day a second run times itself.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — instructions and target document go whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT. Granted by us because we wrote every byte of it — the generator is in the kit and its output is a pure function of the seed. Nothing is derived from a third-party corpus, so there is no upstream licence to honour and no attribution owed. Verified 2026-08-14 by rebuilding from a clean tree and diffing byte for byte (tools/build_corpus.py --verify). Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineDiff against the expected document

Is the file now exactly what it should be — and if not, which lines moved that nobody asked to move?

MethodHow the test was run

evals/score.py, in-process, no key. The expected result is written by the corpus generator at build time, so both the edit and the refusal populations are scored against bytes authored before any system ran.

The inputOne real row, seen by every grader

The formulaWhat it computes

norm(produced) == norm(expected), where norm strips trailing whitespace per line and at end of file and nothing else. Collateral is a unified diff against the EXPECTED document, not against the input — diffing the input would count the requested change itself as damage.

IterationsFour rulers, four wrong numbers

The tell was the ranking inverting, not the value moving. Noise moves a number; an artifact reorders the comparison the page exists to make.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: this grader, against expected documents authored at corpus build time — not against another model. Both the edit and refusal populations are scored against bytes that existed before any system ran.

How tight can the band be? There is no threshold to tune — the grader is a diff and has no knob. What has a denominator worth stating is each refusal family: 8 rows, so ONE row moves a family rate by 12.5 points.

Cadence: Every paid run, and on any change to src/prompt.py or to the refusal families in src/corpus.py. Both change what is being asked, so neither can be compared across the change.

The decisionWhen to reach for it