The business caseThe problem this solves

A configured operating parameter -- a lead time, a safety margin, a service target -- is set once and then trusted, with nobody re-checking it against what the system has actually been doing since. Working out by hand whether reality has drifted away from the number on file means pulling up each parameter's own history, doing the arithmetic, and deciding whether the gap is big enough to matter -- for every parameter, on whatever cadence somebody remembers to do it. Someone manually comparing a configured operating value against a dashboard of observed behaviour, parameter by parameter, on whatever cadence they remember to check.

The inputThe actual parameters

The corpus is 60 parameters, 0.03 MB (csv 1 · jsonl 2). A real configured-value-vs-observed-behaviour comparison names a real supplier, a real committed schedule and a real demand history -- exactly the material an operator will not let leave the building. There is no public corpus of paired (configured value, observed rolling window, adjudicated correction) for the same reason there is no public corpus of production alert histories. Three categories were chosen deliberately -- mean-based, std-based, and mean-based-but-bounded -- to give three genuinely independent correction formulas to exercise in one corpus.

{"parameter_id": "PD-0001", "category": "lead_time", "entity": "Lead time -- Basalt Fleet Co, SKU 53786", "configured_value": 4, "unit": "days", "window_periods": 10}
{"parameter_id": "PD-0002", "category": "lead_time", "entity": "Lead time -- Nordbridge Carriers, SKU 11030", "configured_value": 14, "unit": "days", "window_periods": 10}
{"parameter_id": "PD-0003", "category": "lead_time", "entity": "Lead time -- Ferrous & Kite, SKU 40160", "configured_value": 13, "unit": "days", "window_periods": 10}
{"parameter_id": "PD-0004", "category": "lead_time", "entity": "Lead time -- Basalt Fleet Co, SKU 40282", "configured_value": 4, "unit": "days", "window_periods": 10}
{"parameter_id": "PD-0005", "category": "lead_time", "entity": "Lead time -- Quillon Transit, SKU 54564", "configured_value": 10, "unit": "days", "window_periods": 10}
{"parameter_id": "PD-0006", "category": "lead_time", "entity": "Lead time -- Marrow Logistics, SKU 78989", "configured_value": 9, "unit": "days", "window_periods": 10}
{"parameter_id": "PD-0007", "category": "lead_time", "entity": "Lead time -- Quillon Transit, SKU 99362", "configured_value": 8, "unit": "days", "window_periods": 10}
{"parameter_id": "PD-0008", "category": "lead_time", "entity": "Lead time -- Coastal Intermodal, SKU 12617", "configured_value": 7, "unit": "days", "window_periods": 10}
{"parameter_id": "PD-0009", "category": "lead_time", "entity": "Lead time -- Coastal Intermodal, SKU 35924", "configured_value": 11, "unit": "days", "window_periods": 10}
{"parameter_id": "PD-0010", "category": "lead_time", "entity": "Lead time -- Ferrous & Kite, SKU 91795", "configured_value": 10, "unit": "days", "window_periods": 10}

The outcomeWhat a good result looks like

A FLAG, with a proposed corrected value computed by code and never asserted by the model, or a HOLD -- for a person to review. Decision-free by design; nothing here changes a configured value automatically.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• A parameter whose configured value has plainly and consistently diverged from observed reality — either the floor or the fast-tier model
  Both catch the easy, sustained cases -- 100% recall from the fast tier on lead_time and service_target's clean_drift group, and the floor agrees on every one.
• A slow, still-moving drift the floor's single threshold cannot see — the fast-tier model, reading the trend fact
  3 of 4 lead_time slow_creep parameters were correctly flagged by the model reading the computed trend, which the floor's single mean-deviation threshold missed.
• Getting the exact corrected NUMBER right on a slow-moving drift — a person, not this kit's formula
  The whole-window mean formula lands between where a slow_creep parameter started and where it now is -- 3 of lead_time's 4 slow_creep cases disagreed with gold on exactly this basis.

And where nothing here is good enough:

• A buffer-style parameter (safety_margin) that is actually fine — neither, as built -- add a human second look
  Every one of the 7 genuinely-fine safety_margin parameters was false-flagged by the fast-tier model, which compared the configured buffer to the observed demand LEVEL rather than its variability.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-18. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-18 — r001-param-drift. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — A fresh clone with NO key configured reproduces the corpus byte-identically (python3 tools/build_corpus.py), scores the free floor (python3 -m evals.baseline) and serves the whole panel with a live floor verdict (python3 -m src.app). Measured on 2026-08-18 with API_KEY blank: three commands, all exit 0, plain python3, no install step -- requirements.txt is deliberately empty. What it cannot do without a key is re-run the eval or press Check; the recorded runs render from the committed result files either way.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. END-TO-END per parameter: one HTTP request carrying the assembled prompt, and the reply parsed to a verdict and a reason. Cutting the window and computing its trend/outlier facts happen before this clock starts and cost no network at all -- they are pure local code over a CSV.

Current processWhat it replaces

Someone manually comparing a configured operating value against a dashboard of observed behaviour, parameter by parameter, on whatever cadence they remember to check.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureEvalsEvalsEvalsEvalsPromptPromptPromptPromptArchitectureArchitectureApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• jsonl
• csv
• 3 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one window is

One fixed ten-period rolling window per parameter, cut from data/readings.csv by src/aggregate.py::cut. No train/test split -- every parameter is judged once.

SetupWhat the setup figure measured

There is no index. 0.03s is tools/build_corpus.py generating 60 parameters and 600 readings from a fixed seed, with no clock read and no model called.

LicenceLicence

MIT -- this repository's own licence. Nothing is fetched and no third-party grant is involved, because every parameter and reading is invented here.

Bring your ownBring your own parameters

Point tools/build_corpus.py at your own parameter categories, configured values and observed-reading source, and delete the synthetic generator. src/aggregate.py, src/formulas.py and src/prompt.py read parameters and readings by id and do not care where they came from -- but each new category needs its own threshold and corrected-value formula added to src/formulas.py first.

⚠︎ And what stops being true when you do: The measured recall/precision/value-agreement figures are properties of THIS corpus's three categories and four scenario shapes. A category with a drift shape this corpus does not plant -- a widening spread, two parameters moving together -- has not been measured, and the safety_margin category's own weak precision (43.8%) is evidence that the model does not automatically generalise its judgement across categories with different correction arithmetic.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — but they are the input total apportioned by each part’s share of the characters, not a per-part reading from the provider’s counter. The total is measured; the split is arithmetic, and a part carrying denser text really costs more of it than this says.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 622 tokens assembled, 492 are instructions — 79% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Replayed from the kit's own src/prompt.py::build on the same parameter and window r001-param-drift judged -- the identical code path the run used to assemble it, not a paraphrase. The part character counts below match what evals/run.py recorded for this call.

Full promptThe prompt in full

You are the parameter-review step in a monitoring system. You are shown one configured operating parameter -- a lead time, a safety margin, or a service target -- alongside the system's own rolling window of what it has actually observed for that parameter, plus two facts computed from that window: the trend across the window (first half vs second half) and any outlier readings, each with whatever note was logged against it. Decide whether this parameter should be FLAGGED for a person to review, or HELD. Return a JSON object with exactly these keys: {"verdict": "FLAG" or "HOLD", "reason": "<one sentence, citing the specific fact you relied on>"}.

Parameter PD-0001 -- Lead time -- Basalt Fleet Co, SKU 53786
Category: a configured lead time (a duration)
Configured value: 4 days

Observed window, oldest to most recent:
  t-9          5.7
  t-8          5.9
  t-7          5.8
  t-6          5.5
  t-5          6.8
  t-4          5.6
  t-3          6.0
  t-2          6.0
  t-1          6.2
  t-0          7.0

Trend: first-half mean 5.94, second-half mean 6.16 -- no material trend (3.7% change).
Outliers: none -- every reading is within 2.5 standard deviations of the window mean.

How to decide:
- START WITH THE BASIC COMPARISON: does the configured value match what most of the window actually shows? If the readings CONSISTENTLY sit well away from the configured value -- even with no trend and no outlier, even if the gap has been stable the whole window -- that IS a real, sustained mismatch and should be FLAGGED. 'Stable' is not a reason to hold; a parameter that has been wrong the same way for the whole window is exactly the case this exists to catch, and it is usually the easiest call to make.
- The trend and outlier facts are for the HARDER cases, not a replacement for the basic comparison above: a steady one-directional trend across the window can be worth flagging even when the whole-window average still looks mild, because the average hides a trend that is still under way -- read the trend fact, not just the average, in that case. And a single unusual reading with a logged one-time-event note is NOT a sustained mismatch, however large that one reading is, when the rest of the window otherwise matches the configured value -- HOLD that case specifically.
- If the window's readings are close to the configured value with only ordinary noise, HOLD.
- This is decision-free: nothing you say changes anything automatically. FLAG means 'a person should look at this', not 'change it now'.
Answer with the JSON object only.

JSON:

Raw responseThe raw response

{"verdict": "FLAG", "reason": "All 10 observed readings (5.5-7.0 days) are consistently and substantially above the configured 4-day lead time, showing a sustained mismatch across the entire window."}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch a retailer's out-of-date lead times and safety stock — 60 parameters. Two tiers of one model family answered, and every answer was then graded one way — by == against a fixed gold value, which is a measurement too and only as good as the gold behind it.

The rulerHow it was graded

The grader is evals/scoring.py and it is pure code: the model's FLAG/HOLD verdict against the parameter's gold label produces one of five outcomes, and for a correctly-flagged drift parameter, whether the code-computed corrected value agrees with the gold corrected value (within 15% relative or 0.5 absolute, whichever is larger) is a second, independent check. There is no judge model and no rubric. The same module scores the free floor.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran both scored evals is kept out of these tables per this estate's naming rule, so nothing here is what was actually paid -- the real spend for this kit's build (well under $0.05 across both tiers and the free baseline) is recorded in the commit history, not on this page.

Grading unitWhat the grading figure prices

Grading calls nothing -- a verdict is compared to a label and a computed number is compared to a gold number, both by code. A measured $0.00, not an unpriced one.

The gradersOne way to grade, and why it is the only one

The floor trails the fast tier on recall (59.3% vs 88.9%) because it cannot read the trend or outlier facts -- but its value-agreement rate (68.8%) is close to the fast tier's (70.8%), because both use the identical corrected-value formula; the floor's only difference there is which (smaller) set of correctly-flagged rows it counts among.

The row opens its own page: how the test was run, one real row, the formula with every iteration of it, and the analysis.

Choosing oneWhen to reach for the model at all

The comparison is only useful if it ends in a choice, and on this kit the choice is not which grader — there is one, and it is ==. It is whether to call a model for this field at all, and the answer is conditional.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The baseline (evals/baseline.py) and the scorer (evals/scoring.py) are both pure code and cost $0.00 to run against either result set. Figures above are projected onto the same cards as cost_by_model, not the real spend.

Cost driversWhat actually moves the bill

• The rules block, the single largest prompt part at 1362 of about 2549 characters (~53%) -- the fixed instructional text costs more per call than this specific parameter's own data does.
• Which model: the reasoning tier costs about 8x the fast tier per call for WORSE measured recall, precision and value agreement on this run -- a strictly worse pick here.
• Reasoning is OFF for every call on both tiers (thinking disabled) -- output averaged under 60 tokens for a one-sentence reason on either model, none of it a hidden reasoning pass.

Your volumeWhat it costs at your volume

Linear in parameters and cheap on the fast tier: 60 parameters cost about $0.0052 projected onto this card, so ten times the set is about $0.052. Each call is independent and self-contained -- no shared context or retrieval step to amortise, so nothing here changes shape at higher volume.

Model choiceWhy this model

It is the tier the shared key is pointed at, and it is the cheap one -- the right place to start a question whose answer might be 'the free floor already wins part of this', which on safety_margin it very nearly does.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

Every fast-tier model number on these pages: 60 parameters, one call each.

The codeEvery file, and what it is for

Nine modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Four of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

tools/build_corpus.pycorpus generator — a swap seam

Generates 60 parameters and 600 readings across three categories from a fixed seed. Decides a TRUE underlying value at generation time and samples the window around it with noise -- the gold corrected value is that true value, never run through the pipeline's own formula, so value agreement is a measured question rather than a tautology.

You change it to: Point it at your own parameter shape and your own categories. The pipeline downstream reads parameters, readings and gold by id and does not care where they came from.

# Generate the whole corpus from a fixed seed. No network, no third-party data.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
SEED = 20260818
WINDOW_PERIODS = 10
Z_SAFETY = 1.65
CATEGORIES = ("lead_time", "safety_margin", "service_target")
TRAPS = ("clean_drift", "clean_hold", "one_off_spike", "slow_creep")
TRAP_COUNTS = {"clean_drift": 5, "clean_hold": 7, "one_off_spike": 4, "slow_creep": 4}
SUPPLIERS = ["Halden Freight", "Marrow Logistics", "Coastal Intermodal", "Ferrous & Kite",

src/aggregate.pythe cut

SEAM 2 -- groups the flat readings stream into one ordered window per parameter and computes mean, sample std, a first-half-vs-second-half trend and any outliers, with whatever note the corpus logged against them. Pure code, no model.

# SEAM 2 -- the cut. Grouping raw readings into the rolling window a parameter is judged on, and

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
READINGS = os.path.join(HERE, "data", "readings.csv")
OUTLIER_K = 2.5
def load_readings(path=READINGS):
def cut(rows):
def stats(window):
def trend(window):
def outliers(window, k=OUTLIER_K):
def facts(window):

src/formulas.pythe rule-and-threshold floor — a swap seam

A fixed deviation threshold per category (20% for lead_time, 32% for safety_margin's required-buffer gap, 3 points for service_target), scored before any model call. Also computes the deterministic corrected value -- the SAME formula regardless of what flagged the parameter.

You change it to: DEVIATION_THRESHOLD_PCT, SAFETY_GAP_THRESHOLD_PCT and SERVICE_GAP_THRESHOLD_PTS are each a stated judgement, not fitted to this corpus -- change them and re-run evals/baseline.py, which costs nothing.

# The rule-and-threshold floor, and the deterministic corrected-value calculator. Pure code, no

DEVIATION_THRESHOLD_PCT = 20.0
SAFETY_GAP_THRESHOLD_PCT = 32.0
SERVICE_GAP_THRESHOLD_PTS = 3.0
Z_SAFETY = 1.65
def required_buffer(std):
def deviation(category, configured, window_stats):
def decide_floor(category, configured, window_stats):
def corrected_value(category, window_stats):

src/prompt.pythe prompt

The FLAG/HOLD vocabulary, the trend/outlier facts formatting and the rules text, declared once and read from here by the prompt, the parser and the app.

# SEAM 3 -- the verdict vocabulary and the prompt. Declared once, here, and read from here by

VERDICTS = ("FLAG", "HOLD")
MEANS = {
CATEGORY_LABEL = {
MAX_TOKENS = 4096
SYSTEM = (
RULES = (
def _fmt_window(window):
def _fmt_trend(trend):
def _fmt_outliers(outliers):

src/adapters/__init__.pythe model adapter — a swap seam

SEAM 1 -- the model. Raw HTTP for every provider, stdlib only. Returns finish_reason and reasoning_chars beside the text.

You change it to: .env plus the same run again. Any OpenAI-compatible provider or Anthropic.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

src/triage.pythe AI layer

Assembles the prompt, calls the model, parses the reply, and always computes the deterministic corrected value alongside it -- whether or not anything flagged the parameter.

# SEAM 1 -- the AI layer. Loads the corpus, builds one prompt per parameter, calls the model,

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
def load_parameters():
def load_labels():
def parameters_by_id():
def labels_by_id():
def windows():
def check(cfg, param, window, complete=None, thinking=None):

src/decide.pythe outcome vocabulary

Five outcomes plus a separate value-agreement axis, kept apart. Returns no single accuracy figure.

# The outcome vocabulary. Pure code, no model -- shared by the app, the baseline and the real run

OUTCOMES = ("correct_flag", "correct_hold", "false_flag", "missed_drift", "no_verdict")
MEANS = {
VALUE_OUTCOMES = ("value_agrees", "value_disagrees")
VALUE_TOL_PCT = 15.0
VALUE_TOL_ABS_FLOOR = 0.5
FLAG, HOLD = "FLAG", "HOLD"
def outcome(gold_label, verdict, replied=True):
def value_agrees(gold_value, proposed_value):

evals/baseline.pythe free floor harness

Scores src/formulas.py::decide_floor over all 60 parameters through the identical scorer the model is scored by. 0 calls, $0.00.

# The free floor: `src/formulas.py`'s rule-and-threshold decision, scored through the exact same

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
def main():

src/app.pythe app

Static files and a handful of JSON endpoints on the standard library. Every panel except one button is computed locally, so the product works with no key.

# The minimal local UI. Standard library only -- python -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
PORT = int(os.environ.get("PORT", "8782"))
def _corpus():
class H(BaseHTTPRequestHandler):
def main():

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 630 input and 58 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

No untrusted external input surface. Every fact in the prompt -- the configured value, the observed window, the trend and outlier facts -- is computed directly from this kit's own generated corpus (tools/build_corpus.py, src/aggregate.py); nothing arrives from outside the codebase the way an inbox message or an ingested document does on sibling kits.

The provider key lives only in .env (gitignored) or the shared repo-root .env, read by src/config.py and never logged. src/app.py strips api_key and base_url out of any exception message before it reaches the browser, so a misconfigured key cannot leak into a UI error.

The experimentWe did not attack it -- there is no surface to attack

An indirect prompt injection needs a field an outside party controls that reaches the prompt. This kit has none, so the four rows below are boundaries confirmed by reading the code, not payloads run through it. Confirmed by reading the code, not by a run, on 2026-08-18 -- no attack run was fired; see the experiment section for why.

Each boundary above was checked by reading the call sites, not by an attack trial -- there is no untrusted field to construct a payload against (see posture).

The result0 attack trials, by design -- but 4 boundaries around the model's reply hold, confirmed by reading the code.

This kit has no field an outside party controls that reaches the prompt -- the configured value, the observed window and the trend/outlier facts are all computed from data tools/build_corpus.py generated, never from correspondence, a document, or any other externally supplied text. A future version that ingests real observed readings from a live external system would reopen this question and should be attacked before shipping.

HonestyWhat this does not prove

• Whether a future version of this kit that ingests real observed readings from a live external system (rather than a generated corpus) would reopen an injection surface. Not applicable to the shipped version; worth re-examining if this kit is ever pointed at a live data feed.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

FLAG only surfaces a parameter for a human to review, with a proposed corrected value attached. No code path in this kit changes a configured value automatically.

src/app.py's /api/check endpoint and evals/run.py both return a verdict and a proposed value; neither writes to data/parameters.jsonl or any other store. There is no apply/commit path in this kit at all.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• IT IS NOT A RATE LIMIT, A POLICY CHECK, OR A RUNTIME GUARD CATCHING A BAD MODEL OUTPUT BEFORE IT APPLIES. It is the absence of an apply/commit code path, full stop -- there is no 'undo' because there is nothing here that could 'do' in the first place.
• It does not make the FLAG/HOLD judgement itself correct -- see Eval.taxonomy's false_flag finding on safety_margin, which this guardrail does nothing to prevent.
• It does not validate the proposed corrected value before a person sees it -- a value that disagrees with what a real analyst would approve (see Eval's value_disagrees cases) is shown exactly as computed, with no sanity check.
• A future version wired to a real system would need to add a real, separate approval gate before writing anything; see add_first.

WatchedWhat is watched, and why that one

The kit measures 22 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

12 measured by the latest run 10 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (n_drift 27, n_no_drift 33, parameters 60, thinking False) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

2 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked.

not a time series No two of these 2 runs measured the same system — they differ on the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

DeviationsWhat deviated

0 breaches across 2 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-run evals/baseline.py (free) on any threshold change in src/formulas.py. Re-run evals/run.py (paid, one call per parameter) on any change to src/prompt.py, src/formulas.py or src/aggregate.py.

In one lineNo framework, deliberately

This kit has no framework and no dependencies at all -- csv, statistics, urllib and http.server from the standard library. requirements.txt is an empty file with a comment explaining that the emptiness is load-bearing.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph. The pipeline has one path per parameter -- cut, floor, prompt, call, parse, compute value, score -- with no branching and no state carried between parameters. A framework would add an orchestrator to a for-loop.

The other sideWhat a framework costs you

• A new parameter category needs its own threshold and corrected-value formula hand-written in src/formulas.py rather than configured declaratively.
• There is no built-in retry/backoff beyond src/adapters.py's own bounded retry -- a framework's queue and worker model is not here, so a real deployment adds its own scheduling.
• No built-in observability beyond what evals/run.py prints and writes to results/ -- a framework's tracing/dashboard integration is not here.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-param-drift on the fast tier, 2026-08-18. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 1 earlier run on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — windows are cut from the log stream per run.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT -- this repository's own licence. Nothing is fetched and no third-party grant is involved, because every parameter and reading is invented here. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe model's verdict against the parameter's gold label, and the computed value against gold

Did the model's FLAG/HOLD verdict agree with what the corpus knows that parameter to be, and -- for a correctly flagged parameter -- does the code-computed corrected value agree with the gold corrected value?

MethodHow the test was run

evals/scoring.py::score, in-process, no key and no model. The same function evals/baseline.py calls.

The inputOne real row, seen by every grader

The formulaWhat it computes

recall = correct_flag / (correct_flag + missed_drift); precision = correct_flag / (correct_flag + false_flag); value_agreement = value_agrees / (value_agrees + value_disagrees), counted only over correct_flag rows.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The corpus's own labels and gold corrected values, decided by tools/build_corpus.py at generation time from a TRUE underlying value never run through the pipeline's own formula. This grader IS the reference, so its own TPR/TNR are not separately measured -- that would be circular.

How tight can the band be? There is no tuned threshold on the model path: its verdict decides. The floor's thresholds are stated constants in src/formulas.py, not fitted to this corpus after the fact.

Cadence: Re-run on any change to src/prompt.py, src/formulas.py, src/decide.py or the corpus generator. The free half (evals/baseline.py) costs nothing and should be re-run on every change to src/formulas.py or src/aggregate.py.

The decisionWhen to reach for it