Home › Use Cases › Decide when an online store's server logs should wake someone

Decide when an online store's server logs should wake someone

Pager rules count errors, so they miss the quiet signs of trouble: a service that stops logging, or a warning that a certificate is about to expire. This app reads each five-minute window of logs and decides whether it should wake someone.

For the on-call engineerCross-domain

Why it matters

Today's manual process, and the same job with the app

An on-call engineer at an online store, carrying the pager overnight for its orders, checkout, payments and login systems.

✕Today's manual process

1Pager rules fire when the count of errors passes a set number, or an alarm word appears.

2A broken health check fails every few seconds, does no harm, and wakes someone night after night.

3A quiet warning about an expiring certificate, or a service gone silent, wakes nobody.

4One missed warning and every login stops two days later, and a customer finds it first.

The rules miss the quiet warnings

✓With the app

1Each five-minute window is read, with repeated lines folded into one and any silent service noted.

2The loud, harmless error still wakes someone too often. This is where the app is weakest.

3Quiet warnings are caught: every service gone silent, and most warnings like an expiring certificate.

4Fewer outages are missed, but it is not ready to run the pager on its own.

Quiet warnings caught, loud noise still pages

See it work

One real case: what the app reads, step by step

The store's database runs out of disk space at 04:55, and orders, billing, checkout and payments all lose their connection to it.

Decide when an online store's server logs should wake someoneReference appBuilt to be shaped to your process

1One five-minute window the store's logs from 04:55, before the check runs.
2The database goes down two fatal lines, a second apart: out of disk space.
3Repeats folded into one line each repeating error shows once, with its count.
4Every service behind it fails orders, billing, checkout and payments cannot connect.
5Routine lines in between an order accepted, a settlement queued: not part of the failure.

For engineers

How it is built, and how we measured it

All fourteen steps of the build are written up, from the business case to running it in your own environment.

Kit overview →

15 of 20real incidents that woke someonemeasured in 06 Evals →

25 of 40wake-ups that were for nothingmeasured in 06 Evals →

6 of 6windows with a silent service, caughtmeasured in 01 Business case →

3.8¢to judge all 123 windowsmeasured in 07 Unit cost →

The build, step by step

14 steps

06 Evals · the graders

Make it yours

What you see is a reference app. We shape it to how you work.

Every part of it is built to change, and none of it means starting over.

Your rulesWhat should wake someone, what can wait until morning, and who owns each service.

Your logsThe logs your own services already write, cut into five-minute windows.

Your systemsIts decision goes to the paging tool your on-call team already carries.

Your screensYour service names, the fields your engineers read first, and your own wording.

Want this for your team?

Talk to us

We can run this on your own logs, with your own services and pager rules, inside your environment.

Talk to us →

Home › Use Cases › Decide when an online store's server logs should wake someone

Use caseUC0012

🧪 Use-case kit · runnable

Decide when an online store's server logs should wake someone

A small, forkable project that does one job end to end. Run once for real, and every figure on these pages captured from that run.

The business caseThe problem this solves

Every team running anything has a pager, a pile of rules that decide when it fires, and nobody left who remembers why the rules are what they are. The rules are an error count and a keyword, and they are wrong in both directions at once: a health check that has been failing every six seconds for nine days puts 47 errors into a window and wakes somebody who can do nothing about it, while one WARN line saying a certificate expires in 47 hours wakes nobody — and in two days every login stops. The threshold-and-regex alerting rules somebody wrote once and nobody has re-measured since.

Audience

Anyone who carries a pager, or who owns the rules behind one. The services differ between companies; the shape does not. Every number on these pages came from one real run of this code, not from a vendor page.

The inputThe actual windows of logs

The corpus is 240 windows of logs, 0.93 MB (csv 1 · jsonl 1). ⚠︎ REAL PRODUCTION LOGS CANNOT BE PUBLISHED AT ALL, and that is not a licence technicality. They carry hostnames, internal service names, customer and session identifiers and the shape of somebody's private network — and the interesting lines, the ones an incident turns on, are precisely the ones carrying the most identifying detail. Scrubbing hard enough to make a stream publishable removes the thing this kit measures. There is no public corpus of labelled production alerts for the same reason. So a synthetic stream is not the convenient choice here; it is the only kind of alert-triage corpus a stranger can clone and re-run without signing anything.

The corpus

The 240 windows of logsgenerated from a fixed seed, so no real record, person or institution appears in it.
Where each came fromwritten for this kit rather than collected — the corpus is generated in the kit's own repository, so there is no third-party data in it.

Swap this folder for your own material and the kit is pointed at your windows of logs. That is the whole change — there is no database to migrate.

One windows of log, as the model receives itevents.csv · 1 of 240

ts,service,level,message
2026-03-14T00:00:08Z,checkout,INFO,payment intent 7856 created
2026-03-14T00:00:17Z,billing,INFO,credit note 84289 issued
2026-03-14T00:00:28Z,payments,INFO,refund 47550 completed
2026-03-14T00:00:34Z,postgres,INFO,"checkpoint complete, 4717 buffers written"
2026-03-14T00:00:37Z,auth,INFO,logout for session 10486
2026-03-14T00:00:47Z,payments,INFO,settlement batch 93336 queued
2026-03-14T00:00:48Z,billing,INFO,credit note 98072 issued
2026-03-14T00:00:48Z,search,INFO,query 70608 served in 67ms
2026-03-14T00:00:49Z,postgres,INFO,connection 35354 authorised
2026-03-14T00:00:50Z,billing,INFO,subscription 31999 renewed
2026-03-14T00:00:52Z,billing,INFO,credit note 68891 issued
2026-03-14T00:01:01Z,auth,INFO,logout for session 63458
2026-03-14T00:01:02Z,billing,INFO,dunning run 42194 complete
2026-03-14T00:01:08Z,search,INFO,"suggest cache warm, 14139 entries"
2026-03-14T00:01:21Z,auth,INFO,token issued for session 40550
2026-03-14T00:01:22Z,billing,INFO,invoice 37578 rendered in 240ms
2026-03-14T00:01:28Z,checkout,INFO,cart 3478 abandoned after 130ms
2026-03-14T00:01:33Z,orders,INFO,inventory check 9760 passed
2026-03-14T00:01:33Z,search,INFO,query 85846 served in 174ms
2026-03-14T00:01:45Z,postgres,INFO,connection 90481 authorised
2026-03-14T00:01:49Z,billing,INFO,dunning run 19799 complete
2026-03-14T00:01:52Z,payments,INFO,charge 98348 authorised in 20ms
2026-03-14T00:01:54Z,auth,INFO,login succeeded in 302ms
2026-03-14T00:02:07Z,orders,INFO,inventory check 1854 passed
2026-03-14T00:02:11Z,postgres,INFO,autovacuum finished on table 24902
2026-03-14T00:03:06Z,search,INFO,"suggest cache warm, 63282 entries"
2026-03-14T00:03:11Z,payments,INFO,webhook 24045 acknowledged
2026-03-14T00:03:15Z,deploy,INFO,artefact 45960 published

Abridged — the file continues.

The outcomeWhat a good result looks like

A five-minute window is judged in about 3.5 seconds and either wakes somebody or does not, with the two failure directions counted apart and never averaged.

And when it cannot

⚠︎ ON THIS RUN THE MODEL LOST TO THE FREE RULES AND THAT IS THE PUBLISHED HEADLINE. It missed 5 incidents against the floor's 8 — better — and sent 25 false pages against the floor's 0, which is worse in the direction that gets a pager switched off. 24 of those 25 false pages are the flapping health check: the single loudest and most harmless thing in the corpus.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

Loud, repetitive noise you already recognise — flapping probes, retry storms — the rule engine you already have
It gets every one of them right for nothing, and the model got 24 of 40 flapping windows wrong.
One quiet line that means an outage in two days — a model, on the windows the rules cannot grip
No count reacts to one line and no regex matches ordinary English about a certificate. The model caught 5 of 8.
A service that has stopped logging — code to detect the absence, then a model to judge it
A regex cannot match an absence. Once src/window.py states the fact, the model was right on 6 of 6.
A whole pager, end to end — rules where there is evidence, model where there is none
Measured on this corpus for $0.00 by re-scoring the recorded verdicts: 3 missed and 0 false pages, beating the model (5/25) and the rules (8/0) on BOTH numbers, at 41 calls instead of 123.

At a glanceHow the whole thing runs

5%missed incidents

3,544 msp50, end to end

$0.31per 1,000 windows · Google Gemini 2.5 Flash-Lite

Run once, for real, on 2026-08-13. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Decide when an online store's server logs should wake someone14 steps · 4 questions · run once, for real · 2026-08-13

Does it work?evidence first

STEP 01Business case100.0% answered STEP 03App UI3 shown, 2 failing STEP 06Evals6 of 123 scored STEP 13Observability3,544 ms median call

How is it built?the run, in order

STEP 02Architecture5 swap seams STEP 04Corpus240 windows of logs STEP 05Prompt1,280 tokens in STEP 08Implementationruns with no key STEP 12Frameworksnone, deliberately STEP 14Your environment1 outbound call site

What will it cost?on your rate card

STEP 07Unit cost$0.31 per 1,000 STEP 09Cost modelat your volume

What could go wrong?stated, not hidden

STEP 10Threat model8 of 90 attacks landed STEP 11Guardrailsenforced in code STEP 11Monitoring1 run recorded

Every tile is a linkall 14 steps · no step without a page

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

What do I have to bring?	Point src/window.py at your own CSV of ts,service,level,message and delete tools/build_corpus.py. Corpus lens →
When is this the wrong choice?	Avoid: Paying per window to re-derive a judgement a threshold already makes correctly. That is the case against the best-fitting scenario (“Loud, repetitive noise you already recognise — flapping probes, retry storms”). 4 scenarios scored in all, each with its own. Eval lens →
Where does it stop working?	Windows whose lines do not collapse. The corpus's noise repeats, so 47 identical timeouts become one line; a real stream with request ids or timings in every message collapses far less and the prompt grows accordingly. 4 recorded failure modes, each from a run rather than a guess. Corpus lens →
What was never verified?	Only the 123 windows the gate passed were judged, not all 240. Every score is conditional on the gate having shown the window to the model at all — gate recall is 1.0 here and is published beside every figure for exactly that reason. 5 items this kit says it could not check. Eval lens →
Can I run this on a model I control?	Yes — any OpenAI-compatible endpoint, including one on your own hardware. The shipped adapter takes its host from `BASE_URL` and its model from `MODEL`, so nothing in `src/` changes. The published figures come from 1 model on the fast tier, one provider, one key. Prompt lens →
And if it fits — what do I stand up?	4 artifacts with a stated home and a stated egress, and 3 decisions each with what you provision past its ceiling — plus what was not measured. That is the next page, not this one. step 14 — Run it in your environment →

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-13 — r014-ops-triage-flash. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Run this on your own data

The pipeline, its eval harness and the runs behind every numberdeployed inside your environment, on your own model endpoints, against your own documents.
The corpus above is the shape, not the limitit is a folder swap, and there is no database to migrate.

Talk to us →

Checked before this shipped — A fresh clone with NO key configured reproduces the corpus byte-identically (python3 tools/build_corpus.py), re-derives and checks every label (python3 -m evals.check_labels), scores the free floor over 49 settings (python3 -m evals.baseline) and serves the whole panel with a live threshold (python3 -m src.app). Measured on 2026-08-13 with API_KEY blanked: four commands, all exit 0, plain python3, no install step — requirements.txt is deliberately empty. What it CANNOT do without a key is re-run the eval or press Ask the model; the recorded run renders from the committed result file either way.

Home › Use Cases › Decide when an online store's server logs should wake someone › Architecture

Use caseUC0012

🧪 Lens 02 · Architecture

Architecture

Decide when an online store's server logs should wake someone

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

Corpus Corpus Architecture Architecture Evals Evals Evals Evals Prompt Prompt Prompt Prompt Architecture Architecture App UI App UI Evals Evals Business case Unit cost

1stations numbered in run order
csv
3 of 10 stations carry a recorded failure

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

Sources
csv1
15,189 events, 240 windows, 123 judged
1Sample eventsCorpus ↗
- 15,189 lines, 8 services
- fixed seed, MIT
2Cut into windowsArchitecture ↗
in 5 min
- 20 hours → 240 buckets
- pure code, the unit is made here
3Labelled windowsEvals ↗
- 123 judged after the gate
- 20 page, 103 hold
4Rule-and-threshold floorEvals ↗
- 0 false pages, 8 missed
- 0 calls, free
Recorded failureblind to a certificate expiring
5PromptPrompt ↗
- 1,280 tok
- the window's lines, collapsed
6One modelPrompt ↗
- one key
- 3.5s p50
0.000313 per window · Gemini 2.5 Flash-Lite Unit cost ↗
Recorded failure25 false pages, 24 on one flapping check
7Page thresholdArchitecture ↗
- below it nobody wakes
- pure code
8Paged or heldApp UI ↗
- the decision
- and the lines it turned on
9ScoredEvals ↗
- missed and false, apart
- pure code, free
Recorded failure2 cascades missed — the loudest case
10ReportBusiness case ↗
published, stamped
5missed incidents
25false pages
2026-08-13as of

the free floor already sends 0 false pages and misses 8 of 20 incidents, and the model missed 5 while sending 25. ⚑ THE TWO ARE BLIND IN DIFFERENT PLACES: the model took every silence window and 5 of 8 quiet-killers, which no rule can reach, and fumbled the loud cases a threshold gets for nothing. Rules where there is evidence and a model where there is none scores 3 missed and 0 false pages, at 41 calls instead of 123.

The swap seams

Seam	File	What changes
the model	`src/adapters/__init__.py`	`.env` plus the same run again. Any OpenAI-compatible provider or Anthropic.
the cut	`src/window.py`	The bucket size. ⚠︎ THIS IS NOT A SETTING, IT IS THE QUESTION. Widen it and the retry storm's recovery line falls inside the window, so the correct answer for that window flips from page to hold. The labels are true for a five-minute cut and for no other, so changing it means re-labelling.
the floor	`src/rules.py`	Swap the threshold and regex for Alertmanager, PagerDuty event rules or an anomaly detector. The scorer does not change, so the comparison survives.
the gate	`src/window.py::candidates`	Anything cheaper or broader — but measure gate recall afterwards. It is a ceiling on every score downstream, and the first version of it scored 0.85 by being blind to silence that was already under way.
the decision	`src/decide.py`	A ticket queue in the morning instead of a page tonight, which is a third destination this kit deliberately does not have.

Components

Component	File	Role
corpus generator	`tools/build_corpus.py`	Generates 15,189 events across 8 services from a fixed seed and asserts its own census before writing a byte. No clock is ever read. Its sharpest check is for an UNPLANTED silence — a quiet window where a service happened to emit nothing would be an unlabelled instance of the hardest trap in the set.
the window cutter and the gate	`src/window.py`	SEAM 2 and the first station, doing two jobs. Fixed 300-second buckets turn a stream with no units in it into 240 things that can be judged one at a time, and collapse identical lines so 47 repetitions of one timeout read as one fact carrying its count. Then `candidates()` decides which windows are worth a call — 123 of 240, the other 117 held for free. ⚠︎ THE GATE IS IN THIS FILE AND NOT ITS OWN MODULE ON PURPOSE: it has to compare a window against its history, because the obvious 'any ERROR or WARN' filter drops every silence window and would have deleted this kit's own finding before a model ever saw it.
the rule floor	`src/rules.py`	An error-count threshold plus a real SRE keyword regex — what a great many rotations are running right now. Scored through the same decider as the model, so the comparison is a comparison.
the prompt	`src/prompt.py`	The two verdicts, the rules and the output ceiling, declared once and read from here by the prompt, the parser, the scorer and the app. There is deliberately no third verdict: a pager is a binary actuator.
the model adapter	`src/adapters/__init__.py`	SEAM 1 — the model. Raw HTTP for every provider, stdlib only. Returns `finish_reason` and `reasoning_chars` beside the text, which is how a truncated reply is told apart from a silent one without spending a second run.
the decider	`src/decide.py`	The five outcomes and the two numbers, kept apart. Returns no accuracy figure at all, because on an 8%-page corpus 'hold everything' scores 83.7% while missing every incident in it.
the free floor harness	`evals/baseline.py`	Sweeps the rules over 49 settings across four floors and prints whether the kit's own Stop-A claim survived. 0 calls, $0.00.
the combination harness	`evals/combine.py`	Re-scores the ALREADY-RECORDED verdicts under five combination rules, all declared before any was scored. 0 calls. This is where the architecture that beats both parents came from.
the app	`src/app.py`	Three static files and a handful of JSON endpoints on the standard library. Every panel except one button is computed locally, so the product works with no key.

Where it breaks at scale

Not on volume — windows are independent and the work is linear in them. It breaks on WINDOW SIZE and on cardinality. A busy production stream puts thousands of lines in five minutes; collapsing helps enormously (median 89 raw lines became 52 distinct ones here) but a real system with hundreds of services and high-cardinality messages will not collapse nearly as well, and the prompt grows until the window no longer fits. The honest ceiling is the point at which a window stops fitting in context, and nothing in this kit measures where that is on a real stream.

Home › Use Cases › Decide when an online store's server logs should wake someone › Data

Use caseUC0012

🧪 Lens 04 · Data

Corpus

Decide when an online store's server logs should wake someone

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

240windows of logs

0.93 MiBcsv 1 · jsonl 1

p50 89chars per window

$0.00setup · 2.0s

How it is cutWhat one window is

Fixed 300-second buckets over a 20-hour stream, cut by src/window.py. The gate then passes 123 of the 240 windows to a model and holds the rest for free.

SetupWhat the setup figure measured

There is no index. The 2.0s is tools/build_corpus.py generating the stream from a fixed seed — 15,189 events across 8 services, asserting its own census before writing a byte, with no clock read and no model called. The only reduction step is src/window.py, which cuts 300-second buckets and collapses repeated lines at call time over the stream already on disk, not as a build stage. It costs nothing because nothing is sent.

LicenceLicence

MIT — this repository's own licence, verified against LICENSE at the repo root on 2026-08-13. Nothing is fetched and no third-party grant is involved, because every event is invented here.

Bring your ownBring your own windows of logs

Point src/window.py at your own CSV of ts,service,level,message and delete tools/build_corpus.py. ⚠︎ THE LABELS ARE THE WORK, NOT THE LOGS. Everything this kit measures is a comparison against a human decision about whether each window should have woken somebody, and a real stream arrives with none. Expect to label a few hundred windows before any number here means anything about your system — and to re-label from scratch if you change the bucket size, because the labels are true for one cut only.

What breaks it

Windows whose lines do not collapse. The corpus's noise repeats, so 47 identical timeouts become one line; a real stream with request ids or timings in every message collapses far less and the prompt grows accordingly.
Incidents that do not sit inside one bucket. A cascade split across four windows looks like four separate outages; this corpus spans at most two, which is kinder than reality.
Services that are legitimately quiet. deploy is excluded from the silence check by name because it says nothing for hours and that is health — on a real estate that list is long and nobody maintains it.
A stream with no history yet. gone_silent refuses to answer for the first six windows rather than guessing, so the opening half-hour of any stream is judged without it.

Home › Use Cases › Decide when an online store's server logs should wake someone › LLM

Use caseUC0012

🧪 Lens 05 · LLM

Prompt

Decide when an online store's server logs should wake someone

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

Part	Characters	Tokens
What the model is being asked to be	240	not measured
The window, its lines collapsed	2,879	not measured
How to decide	747	not measured
Total		1,280

This is the cost lesson as arithmetic: of the 3,866 characters assembled, 2,879 are datas — 74% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Replayed from the kit's own evals/run.py::prompt_for on the same window the run judged, which is the identical code path the run used to assemble it — not a paraphrase and not a reconstruction. The recorded row for that window carries prompt_chars, and the replayed text matches it exactly, which is what makes the published prompt checkable rather than merely plausible.

Full promptThe prompt in full

The verbatim prompt, as sent

the prompt, as sent

You are the on-call triage step for a production system. You are shown one five-minute window of logs and you decide whether it should wake a human being right now. Answer with exactly one word: PAGE or HOLD. No explanation, no punctuation.

Window w039, five minutes beginning 2026-03-14T03:15:00Z.

03:15:02  postgres  INFO  replication lag 15ms
03:15:17  billing   INFO  dunning run 34630 complete
03:15:23  postgres  INFO  checkpoint complete, 45846 buffers written
03:15:27  payments  INFO  refund 76536 completed
03:15:32  search    INFO  query 62752 served in 117ms
03:15:49  billing   INFO  dunning run 61341 complete
03:15:51  search    INFO  query 90210 served in 256ms
03:15:56  auth      INFO  logout for session 38359
03:15:57  orders    INFO  order 83206 dispatched in 14ms
03:16:02  auth      WARN  certificate for auth.internal expires in 47 hours
03:16:03  auth      INFO  certificate renewal job last succeeded 62 days ago
03:16:03  payments  INFO  charge 75138 authorised in 145ms
03:16:18  checkout  INFO  session 20191 started
03:16:19  search    INFO  query 64763 served in 79ms
03:16:22  checkout  INFO  cart 36187 converted in 211ms
03:16:30  payments  INFO  charge 81850 authorised in 163ms
03:16:32  postgres  INFO  connection 59482 authorised
03:16:36  billing   INFO  invoice 3919 rendered in 69ms
03:16:41  checkout  INFO  session 74411 started
03:16:42  search    INFO  index segment 79513 merged
03:16:48  search    INFO  index segment 88716 merged
03:16:53  auth      INFO  logout for session 9842
03:17:13  payments  INFO  refund 49977 completed
03:17:14  deploy    INFO  artefact 81550 published
03:17:21  postgres  INFO  checkpoint complete, 43298 buffers written
03:17:22  search    INFO  index segment 87202 merged
03:17:26  billing   INFO  credit note 61255 issued
03:17:27  payments  INFO  refund 11552 completed
03:17:28  billing   INFO  credit note 61323 issued
03:17:38  postgres  INFO  connection 39739 authorised
03:17:44  checkout  INFO  payment intent 53681 created
03:17:50  orders    INFO  order 70635 accepted
03:17:52  search    INFO  query 84408 served in 228ms
03:17:58  postgres  INFO  autovacuum finished on table 87779
03:18:10  auth      INFO  logout for session 97177
03:18:14  deploy    INFO  artefact 45820 published
03:18:14  search    INFO  query 71668 served in 195ms
03:18:17  auth      INFO  logout for session 23259
03:18:19  orders    INFO  inventory check 94032 passed
03:18:30  postgres  INFO  connection 34078 authorised
03:18:38  checkout  INFO  cart 44674 abandoned after 110ms
03:18:39  search    INFO  index segment 45222 merged
03:18:43  orders    INFO  inventory check 61806 passed
03:18:49  auth      INFO  logout for session 39478
03:18:58  deploy    WARN  canary 26369 restarted once before becoming healthy
03:19:00  billing   INFO  dunning run 65056 complete
03:19:08  postgres  INFO  autovacuum finished on table 24038
03:19:22  postgres  INFO  autovacuum finished on table 5651
03:19:23  payments  INFO  settlement batch 88142 queued
03:19:25  orders    INFO  order 83221 accepted
03:19:33  checkout  INFO  cart 76236 abandoned after 87ms
03:19:43  payments  INFO  settlement batch 82557 queued

How to decide:
- PAGE means a person is woken at 3am. Volume alone is not a reason: a check that has been failing identically for days, a retry storm that recovers, and errors that start at a deploy and stop on their own are all noise, however many lines they produce.
- A single quiet line can be an emergency. An expiring certificate, a replication slot filling a disk, a batch nobody acknowledged — nothing is on fire yet and something will be.
- A service that has stopped logging entirely may be down. If the window states that a service emitted nothing while it was talking steadily before, treat the silence as evidence.
- Do not page for something already recovering inside the same window.
- If it can safely wait until the morning, HOLD.

One word:

Raw responseThe raw response

The raw response, before any parsing

the response, unparsed

PAGE

Home › Use Cases › Decide when an online store's server logs should wake someone › Eval

Use caseUC0012

🧪 Lens 06 · Eval

Evals

Decide when an online store's server logs should wake someone

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Decide when an online store's server logs should wake someone — 240 windows drawn from 240 real windows of logs. One model answered, and every answer was then graded one way — by == against a fixed gold value, which is a measurement too and only as good as the gold behind it.

The rulerHow it was graded

The grader is src/decide.py and it is pure code: the model's one word against the window's label, producing one of five outcomes. There is no judge model and no rubric, because there is nothing to interpret — PAGE and HOLD are the whole vocabulary and the label is one of two words. ⚑ THE SAME MODULE SCORES THE FREE FLOOR, which is what makes the comparison a comparison rather than two measurements taken with different instruments.

240windows

240source documents

1model tier

1grading method

MeasurementsWhat was measured

COUNTED6 / 123missed incidents — windows judgedDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.

COUNTED31 / 123false pages — windows judgedDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.

COUNTED15 / 20detection pct — windows that should have pagedDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.

COUNTED15 / 40page precision pct — pages the model sentDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.

NOT YET KNOWN—A person confirmed the gold is rightThe grader is == and needs no confirming. The gold does: it is machine-derived from the registry's structured modules, and every rate on this page rests on that derivation being faithful to what the prose actually says. Nobody has read a document against its gold record by hand.

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

How the method was validated

⚠︎ THE METHOD IS TRIVIAL AND THE LABELS ARE THE THING THAT NEEDED VALIDATING. A two-word comparison cannot be wrong about itself, so the risk is entirely in the ground truth. evals/check_labels.py re-derives every label from the event stream by a different code path from the generator that wrote them, and refuses the set if they disagree — with the silence labels checked specifically, because they are the one kind whose evidence is an absence and can therefore be invalidated by an edit that changes no text anywhere. It passes at 0 problems.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill. ⚠︎ AND ON THIS KIT THE OUTPUT SIDE IS THE UNSTABLE HALF: the average reply carried 462 output tokens for a one-word answer, almost all of it this model's own reasoning, so a card for a model that answers without reasoning would overstate the cost several times over.

Priced at	Per 1M in / out	One window	1,000 windows	Share that is the prompt
Google Gemini 2.5 Flash-Lite the cheapest model Google publishes a rate for, and the card every kit in this series projects onto so the rows compare across use cases	$0.10 / $0.40	$0.000313	$0.31	41%
Amazon Nova Pro on Bedrock a frontier tier, for the other end of the range	$0.80 / $3.20	$0.002502	$2.50	41%
OpenAI GPT-4o mini the mid tier most teams reach for first, and the one a reader is most likely to have a key for already	$0.15 / $0.60	$0.000469	$0.47	41%

Same work, 8× the bill

The same windows, the same tokens — only the rate card changed. And across all 3 cards about 41% of what you pay is the prompt this pipeline sends, not the answer it writes.

⚑ ASK ABOUT FEWER WINDOWS, NOT SHORTER ONES. The prompt is already the window's own collapsed lines and there is little fat in it; the real lever is the gate, and beyond it the escalation rule — sending a model only the windows where a threshold and a regex have nothing to grip cuts the bill to 33% of this run's and IMPROVES both numbers.

Rates checked 2026-08-13. The provider that actually ran r014 is priced from build/facts/models.json, which is the site's own dated model table, and it is the only figure on these pages that is what was actually paid rather than a projection.

Grading unitWhat the grading figure prices

freegrading cost, as measured

Grading calls nothing. A word is compared to a label, so this is a measured $0.00 rather than an unpriced one — and the distinction matters, because a manual method priced in tokens also reads as $0.00.

The gradersOne way to grade, and why it is the only one

⚑ THIS FLOOR IS NOT A STRAWMAN AND IT BEAT THE MODEL ON THE NUMBER THAT GETS PAGERS SWITCHED OFF. 8 missed incidents and 0 false pages, against the model's 5 and 25. ⚑ AND THE KNOB EVERYBODY TUNES IS INERT: deleting the error-count threshold entirely produces the identical result to its best setting — same misses, same zero false pages, same 5 traps — while every other setting of it only adds false pages. That is measured with a no-count control row, not inferred from two tables that look alike.

Grader	Cost / 1k	Data leaves	Same answer twice	Result
The model's word against the window's label Did the model's word — PAGE or HOLD — agree with what the corpus knows that window to be?	$0.00	no	yes	the fast tier 75.0% detection · 1 more measured on each run

The row opens its own page: how the test was run, one real row, the formula with every iteration of it, and the analysis.

Why this set cannot separate them

Yes, and unusually clearly — the two deciders fail on DISJOINT traps, which is what makes this set worth having. The rules are perfect on flapping, retry storms and deploys and blind on quiet-killers; the model is the reverse. ⚠︎ WHAT IT CANNOT SEPARATE IS BUCKET SIZE. Every label is true for a five-minute cut, so this set cannot tell you whether a different cut would change the ordering, and nothing here measures that.

Choosing oneWhen to reach for the model at all

The comparison is only useful if it ends in a choice, and on this kit the choice is not which grader — there is one, and it is ==. It is whether to call a model for this field at all, and the answer is conditional.

If your situation is	Use	Because	And avoid
Loud, repetitive noise you already recognise — flapping probes, retry storms	the rule engine you already have	It gets every one of them right for nothing, and the model got 24 of 40 flapping windows wrong.	Paying per window to re-derive a judgement a threshold already makes correctly.
One quiet line that means an outage in two days	a model, on the windows the rules cannot grip	No count reacts to one line and no regex matches ordinary English about a certificate. The model caught 5 of 8.	Expecting a keyword list to grow to cover this. It is a fuse, not a fire.
A service that has stopped logging	code to detect the absence, then a model to judge it	A regex cannot match an absence. Once src/window.py states the fact, the model was right on 6 of 6.	Asking a model to notice what is missing from its own input. Nobody put it there.
A whole pager, end to end	rules where there is evidence, model where there is none	Measured on this corpus for $0.00 by re-scoring the recorded verdicts: 3 missed and 0 false pages, beating the model (5/25) and the rules (8/0) on BOTH numbers, at 41 calls instead of 123.	Either alone. Each is blind exactly where the other sees.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Cause	What it means	Rows	A real example
`paged_correct`	Paged, correctly	15	A cascade window: postgres FATAL, five services failing on one root cause.
`missed_incident`	Missed incident	5	3 quiet-killers and 2 cascades. The cascades are the alarming ones — the loudest real outage in the corpus, held.
`false_page`	False page	25	24 of the 25 are the flapping health check — 47 identical timeouts that have never once meant anything.
`held_correct`	Held, correctly	78	Ordinary traffic, deploy windows that explain themselves, retry storms that recover inside the window.
`no_verdict`	No verdict	0	None. Every reply was exactly PAGE or HOLD, and `finish_reason` was `stop` on all 123 calls.

What we could NOT verify

Only the 123 windows the gate passed were judged, not all 240. Every score is conditional on the gate having shown the window to the model at all — gate recall is 1.0 here and is published beside every figure for exactly that reason.
The five-minute bucket is a product decision, not a measurement. ⚑ THE FLOOR HALF IS MEASURED NOW, FOR 0 CALLS (evals/recut.py), AND THE KNOB IS EXPENSIVE: re-cut at ten minutes and the free floor's detection falls 24.7 points, at fifteen minutes 24.3, with page precision unmoved at 100%. ⚠︎ AND IT DOES NOT BLUNT THE ABSENCE SIGNAL, IT DELETES IT — gone_silent fires on 6 windows at five minutes, 1 at ten and ZERO at fifteen, because a service quiet for five minutes still logs in the other half of a bigger bucket. The trap this kit was built around becomes undetectable before any model is asked. ⚠︎ THE MODEL HALF IS STILL NOT MEASURED AND CANNOT BE FROM THIS RUN: r014's 123 verdicts were reached on five-minute windows and are not transferable to another cut, which is exactly why window_seconds rides as a comparability guard.
The corpus is invented, so it contains the six failure modes we thought to plant and no others — and its incidents are politely one per window, which real ones are not.
The absence of lines is computed by code and handed to the model as a stated fact. This measures whether a model judges silence correctly ONCE TOLD about it, not whether a model can notice something missing from its input. Those are different questions and only the first was asked.
One tier, one run. A second tier on the identical 123 windows was not run and is not authorised, so nothing here is a band.

Home › Use Cases › Decide when an online store's server logs should wake someone › Cost

Use caseUC0012

🧪 Lens 07 · Cost

Unit cost

Decide when an online store's server logs should wake someone

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

Tier	Input tokens	Output tokens	p50 latency	Google Gemini 2.5 Flash-Lite	Amazon Nova Pro on Bedrock	OpenAI GPT-4o mini
the fast tier	1,280	462	3,544 ms	$0.000313	$0.002502	$0.000469

The token counts are measured on a real run and belong to this pipeline — top_k, chunk size, prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-13. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

One query costs many times as much on a frontier rate card as on the cheapest one, for identical work

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingGrading is free here, and it is a measured zero

One word is compared to one label by src/decide.py. Nothing is called, so this is $0.00 rather than approximately nothing. ⚑ AND THE FREE FLOOR IS FREE ON THE SAME TERMS: 49 settings across four floors, 123 windows each, $0.00.

Cost driversWhat actually moves the bill

⚑ REASONING TOKENS, WHICH ARE MOST OF THE OUTPUT BILL. Output averaged 462 tokens for a ONE-WORD answer, and output is priced at 2.0× input on this model. A model that answers without reasoning would cost a fraction of this.
The window count, which the gate sets. 123 of 240 windows never reach a model at all, and the escalation rule would cut that to 41.
The input prompt, at about 1280 tokens per window — the collapsed lines plus the rules. It is stable because collapsing bounds it, and that is the whole reason collapsing exists.

Your volumeWhat it costs at your volume

Linear in windows and cheap: 123 windows cost $0.037959, so ten times the stream is about $0.3796. ⚑ BUT THE INTERESTING SCALING IS DOWNWARD. The escalation architecture judged the same stream with 41 calls instead of 123 — 33% of the traffic — because most windows have error lines in them and the rules are right about those for nothing. Ten times the stream on that design is roughly a third of the naive bill.

Where pricing changes shape

A window that stops collapsing. The bill is driven by prompt size, and prompt size is driven by DISTINCT lines, not by traffic — a stream that puts a request id in every message does not collapse and the input cost jumps in shape, not in degree.

Your return, with your numbers

VolumeOne five-minute window per five minutes of stream, per system: 288 windows a day, of which about half carry enough signal to reach the gate. This run judged 123.
What it replacesNot a person — nobody reads every window. It replaces the threshold somebody set once, and the cost it is measured against is the false pages that threshold produces and the incidents it misses.
Time saved per item⚠︎ NOT MEASURED, AND NOT MEASURABLE FROM THIS CORPUS. The return on alert triage is an outage caught earlier and a night's sleep not lost, and neither is in a synthetic log stream. What IS measured is the exchange rate: on this run the model bought 3 incidents for 25 false pages, and the escalation design bought 5 for 0.

We publish the inputs, not a return: a return depends on your labour cost and your volume. The ROI dashboard takes it from here, and AI Costing explains the math.

Model choiceWhy this model

It is the tier the shared key is pointed at, and it is the cheap one — which is the right place to start a question whose answer might be 'the free rules already win'. On this corpus that is close to what happened.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

157,449input tokens · this run

56,842output tokens

$0.038what it actually cost

Every model number on these pages: 123 windows, one call each, on the fast tier.

Model	Provider	One eval pass	This whole run	Per 1,000 queries	Rates as of
gpt-5-6-luna	OpenAI	$0.100	$0.100	$0.81	2026-09-12
gemini-3-flash	Google	$0.249	$0.249	$2.03	2026-09-18
gemini-3-8-flash	Google	$0.331	$0.331	$2.69	2026-09-18
llama-5	Meta	$0.438	$0.438	$3.56	2026-09-18
claude-haiku-4-5	Anthropic	$0.442	$0.442	$3.59	2026-09-12
grok-4-5	xAI	$0.656	$0.656	$5.33	2026-09-18
grok-4-6	xAI	$0.656	$0.656	$5.33	2026-09-18
claude-sonnet-5	Anthropic	$0.883	$0.883	$7.18	2026-09-12
gemini-3-1-pro	Google	$0.997	$0.997	$8.10	2026-09-18
gpt-5-6-terra	OpenAI	$0.997	$0.997	$8.10	2026-09-12
gpt-5-6-sol	OpenAI	$1.766	$1.766	$14.36	2026-09-12
claude-opus-4-8	Anthropic	$2.208	$2.208	$17.95	2026-09-12
claude-opus-5	Anthropic	$2.208	$2.208	$17.95	2026-09-12
claude-fable-5	Anthropic	$4.416	$4.416	$35.90	2026-09-18
claude-fable-5-1	Anthropic	$4.416	$4.416	$35.90	2026-09-18
gpt-6-astra	OpenAI	$4.416	$4.416	$35.90	2026-09-17

Read this against the numbers above

⚠︎ NOT MEASURED. Not one of these rows was run. They are this run's measured token counts multiplied by a published rate.
Rates are as of each model's as_of date in build/facts/models.json and nothing checks them against the vendors.
⚑ THE OUTPUT COLUMN IS THE ONE THAT WILL BE WRONG. 462 output tokens for a one-word answer is almost entirely this model's reasoning; a non-reasoning model answers in one or two.
Token counts are not comparable across providers. A sibling kit measured two tiers of one vendor counting a byte-identical prompt as 189-200 against 268-279 input tokens.
Nothing here includes retries, and this run had none — finish_reason was stop on all 123 calls.
⚑ A CHEAPER ROW IS NOT A CHEAPER KIT, AND ON THIS ONE THE FREE OPTION WON A COLUMN. The rule floor costs $0.00 and sent 0 false pages against the model's 25. What money bought was 3 fewer missed incidents — and the escalation design bought 5 fewer for a third of the calls.

Home › Use Cases › Decide when an online store's server logs should wake someone › Implementation

Use caseUC0012

🧪 Step 08

Implementation

Decide when an online store's server logs should wake someone

The codeEvery file, and what it is for

Nine modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

tools/build_corpus.pycorpus generator

Generates 15,189 events across 8 services from a fixed seed and asserts its own census before writing a byte. No clock is ever read. Its sharpest check is for an UNPLANTED silence — a quiet window where a service happened to emit nothing would be an unlabelled instance of the hardest trap in the set.

tools/build_corpus.py

# Generate the event stream and the labelled windows from a fixed seed.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
EVENTS = os.path.join(HERE, "data", "events.csv")
LABELS = os.path.join(HERE, "data", "labelled.jsonl")
SEED = 20260813
START = dt.datetime(2026, 3, 14, 0, 0, 0, tzinfo=dt.timezone.utc)
WINDOW_S = 300
N_WINDOWS = 240
N_QUIET = 150
N_FLAPPING = 40

src/window.pythe window cutter and the gate — a swap seam

SEAM 2 and the first station, doing two jobs. Fixed 300-second buckets turn a stream with no units in it into 240 things that can be judged one at a time, and collapse identical lines so 47 repetitions of one timeout read as one fact carrying its count. Then candidates() decides which windows are worth a call — 123 of 240, the other 117 held for free. ⚠︎ THE GATE IS IN THIS FILE AND NOT ITS OWN MODULE ON PURPOSE: it has to compare a window against its history, because the obvious 'any ERROR or WARN' filter drops every silence window and would have deleted this kit's own finding before a model ever saw it.

You change it to: The bucket size. ⚠︎ THIS IS NOT A SETTING, IT IS THE QUESTION. Widen it and the retry storm's recovery line falls inside the window, so the correct answer for that window flips from page to hold. The labels are true for a five-minute cut and for no other, so changing it means re-labelling.

src/window.py

# SEAM 2 — the cut. Turning a stream into things that can be judged one at a time.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
EVENTS = os.path.join(HERE, "data", "events.csv")
START = dt.datetime(2026, 3, 14, 0, 0, 0, tzinfo=dt.timezone.utc)
WINDOW_S = 300
LOUD = ("ERROR", "FATAL")
ALWAYS_TALKING = ("checkout", "auth", "payments", "orders", "billing", "search", "postgres")
HISTORY = 6
def load_events(path=EVENTS):
def index_of(stamp):

src/rules.pythe rule floor — a swap seam

An error-count threshold plus a real SRE keyword regex — what a great many rotations are running right now. Scored through the same decider as the model, so the comparison is a comparison.

You change it to: Swap the threshold and regex for Alertmanager, PagerDuty event rules or an anomaly detector. The scorer does not change, so the comparison survives.

src/rules.py

# SEAM 3 — the free floor: the rule engine almost every on-call rotation is actually running.

DEFAULT_THRESHOLD = 20
KEYWORDS = re.compile(
def keyword_hits(win):
def decide_window(win, threshold=DEFAULT_THRESHOLD, use_count=True, use_keyword=True):
def score(windows, labels, threshold=DEFAULT_THRESHOLD, use_count=True, use_keyword=True):

src/prompt.pythe prompt

The two verdicts, the rules and the output ceiling, declared once and read from here by the prompt, the parser, the scorer and the app. There is deliberately no third verdict: a pager is a binary actuator.

src/prompt.py

# The verdict vocabulary and the prompt. Declared once, here, and read from here by everything.

VERDICTS = ("PAGE", "HOLD")
MAX_TOKENS = 4096
MEANS = {
SYSTEM = (
RULES = (
def render(win, collapsed, silent=None, history=None):
def parse(reply):

src/adapters/init.pythe model adapter — a swap seam

SEAM 1 — the model. Raw HTTP for every provider, stdlib only. Returns finish_reason and reasoning_chars beside the text, which is how a truncated reply is told apart from a silent one without spending a second run.

You change it to: .env plus the same run again. Any OpenAI-compatible provider or Anthropic.

src/adapters/__init__.py

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
def complete(cfg, system, user, max_tokens=1024):

src/decide.pythe decider — a swap seam

The five outcomes and the two numbers, kept apart. Returns no accuracy figure at all, because on an 8%-page corpus 'hold everything' scores 83.7% while missing every incident in it.

You change it to: A ticket queue in the morning instead of a page tonight, which is a third destination this kit deliberately does not have.

src/decide.py

# The page decision, and the five outcomes it can produce. Pure code, no model.

OUTCOMES = ("paged_correct", "missed_incident", "false_page", "held_correct", "no_verdict")
MEANS = {
PAGE, HOLD = "PAGE", "HOLD"
def outcome(label, verdict, replied=True):
def tally(rows):
def by_trap(rows):
def trap_verdict(rows):

evals/baseline.pythe free floor harness

Sweeps the rules over 49 settings across four floors and prints whether the kit's own Stop-A claim survived. 0 calls, $0.00.

evals/baseline.py

# What you get without a model. Run this BEFORE reading any score this kit produces.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
THRESHOLDS = (1, 2, 5, 10, 16, 20, 25, 31, 40, 47, 48, 49, 54, 60, 100, 220)
TRAPS = ("flapping", "retry-storm", "deploy", "quiet-killer", "cascade", "silence")
def floor_rows(windows, labels, threshold, use_count=True, use_keyword=True, use_absence=False):
def handled(rows):
def main():

evals/combine.pythe combination harness

Re-scores the ALREADY-RECORDED verdicts under five combination rules, all declared before any was scored. 0 calls. This is where the architecture that beats both parents came from.

evals/combine.py

# Score the rules and the model TOGETHER, from outputs already recorded. 0 calls, $0.00.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
FLOOR_THRESHOLD = 49
def load_run(run_id):
COMBOS = [
def main():

src/app.pythe app

Three static files and a handful of JSON endpoints on the standard library. Every panel except one button is computed locally, so the product works with no key.

src/app.py

# The local app. Three static files and a handful of JSON endpoints, on the standard library.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
RESULTS = os.path.join(HERE, "results")
PORT = int(os.environ.get("PORT", "8012"))
def corpus():
def window_rows(limit=None):
def recorded():
def baseline():
class Handler(http.server.SimpleHTTPRequestHandler):

Start hereThe shortest path into it

src/app.pyrun it and click around. It holds no pipeline logic on purpose; every number on screen comes from a module the eval harness already drives.
tools/build_corpus.pyGenerates 15,189 events across 8 services from a fixed seed and asserts its own census before writing a byte. No clock is ever read. Its sharpest check is for an UNPLANTED silence — a quiet window where a service happened to emit nothing would be an unlabelled instance of the hardest trap in the set.
src/window.pySEAM 2 and the first station, doing two jobs. Fixed 300-second buckets turn a stream with no units in it into 240 things that can be judged one at a time, and collapse identical lines so 47 repetitions of one timeout read as one fact carrying its count. Then candidates() decides which windows are worth a call — 123 of 240, the other 117 held for free. ⚠︎ THE GATE IS IN THIS FILE AND NOT ITS OWN MODULE ON PURPOSE: it has to compare a window against its history, because the obvious 'any ERROR or WARN' filter drops every silence window and would have deleted this kit's own finding before a model ever saw it. A swap seam.
src/rules.pyAn error-count threshold plus a real SRE keyword regex — what a great many rotations are running right now. Scored through the same decider as the model, so the comparison is a comparison. A swap seam.
src/prompt.pyThe two verdicts, the rules and the output ceiling, declared once and read from here by the prompt, the parser, the scorer and the app. There is deliberately no third verdict: a pager is a binary actuator.
src/adapters/__init__.pySEAM 1 — the model. Raw HTTP for every provider, stdlib only. Returns finish_reason and reasoning_chars beside the text, which is how a truncated reply is told apart from a silent one without spending a second run. A swap seam.
src/decide.pyThe five outcomes and the two numbers, kept apart. Returns no accuracy figure at all, because on an 8%-page corpus 'hold everything' scores 83.7% while missing every incident in it. A swap seam.
evals/baseline.pySweeps the rules over 49 settings across four floors and prints whether the kit's own Stop-A claim survived. 0 calls, $0.00.
evals/combine.pyRe-scores the ALREADY-RECORDED verdicts under five combination rules, all declared before any was scored. 0 calls. This is where the architecture that beats both parents came from.

Every entry above is a file in this kit, listed in the order the pipeline runs it.

Home › Use Cases › Decide when an online store's server logs should wake someone › Cost model

Use caseUC0012

🧪 Step 09

Cost model

Decide when an online store's server logs should wake someone

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 1280 input and 462 output tokens per query at top-k 1, of which about 150 tokens are fixed (system prompt, chat framing, the question itself) and the rest is retrieved context. Change the retrieval depth and the input scales with it. Change the volume and the bill scales linearly, because nothing here batches.

Querys per day Passages retrieved (top-k) Model Eval technique Rows graded (%)

Grading really is free here, and that is a property of the ruler rather than a discount

This kit is graded by pure code — the grader makes no model call, needs no key and sends nothing anywhere, so the grading line above is zero at every sampling rate and the leak count stays at nought. What that buys is a ruler you can run on every row, every time, for nothing. What it costs is reach: a code grader can only check what is mechanically checkable.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Shape	Querys/day	Per day	Per month	Per year

Model and top-k follow the controls above.

What this ignoresWhere a real bill diverges

Read before quoting any of it

Output length is assumed constant.A terser or more verbose model moves it, and output is priced above input everywhere.
No volume, committed-use, batch or cache discount is modelled.Each moves real enterprise pricing and none are on a public rate card.
Retrieval is free here and stops being free at scale.This kit scores every chunk in Python; a corpus a hundred times larger needs a vector store, which is infrastructure this arithmetic does not price.
Grading adds nothing to this bill, and would if you changed the ruler.A code grader is free and local; swapping in an LLM judge would cost more per row than the extraction itself and send every row to a vendor. The instrument is a cost decision, not a footnote.
Rates age.Every one carries the date it was read on the cost page.

Home › Use Cases › Decide when an online store's server logs should wake someone › Threat model

Use caseUC0012

🧪 Step 10

Threat model

Decide when an online store's server logs should wake someone

In one lineWhat this kit exposes

A local, single-user tool over a corpus you generated yourself. It binds 127.0.0.1, has no auth because it has no second user, and reads a CSV from disk. Nothing is uploaded and no key is ever written into the repo.

Read from the shared .env or the real environment only, never written into the repo, never logged, never returned by any endpoint. /api/status reports whether a key EXISTS as a boolean and nothing more. ⚠︎ AND THE SHARED .env AT THE REPO ROOT MEANS A BRAND-NEW KIT ALREADY HAS A LIVE KEY — 'I have not configured anything yet' is not the same as 'nothing can spend'.

The experimentWe attacked the rules, and named the surface we did not test

⚑ WE ATTACKED THE PAGER'S RULES, NOT THE PROMPT — because on this kit the rules are what stands between a log line and somebody's phone. Six trap families, 123 windows, swept through the free floor at its best setting by evals/baseline.py, with NO MODEL INVOLVED and nothing sent anywhere. The expensive direction is the only one counted below: an incident the rules held. ⚠︎ AND THE SURFACE WE DID NOT MEASURE IS NAMED RATHER THAN OMITTED: a log line reaches the prompt verbatim, so anybody who can make a service write one can put text in front of the model. That is one call per attempt, no attempt was authorised, and a red-team rate nobody ran would be worse than none. Measured on 2026-08-13. The structural facts below are code-checked; the injection rate is explicitly absent.

What you would do	What it does to the pager	What it costs, measured
Trust the rules you already have	Every loud trap correct — flapping, retry storms and deploys all held, cascades all paged. It is not a strawman and it is not wrong about anything it can see.	8 missed incidents and 0 false pages. Structurally blind to a certificate quietly expiring, because there is no alarming word in the line and no count reacts to one line.
Tune the threshold until the noise stops	Feels like progress. The false pages go to zero at 49.	⚑ AND THE MISSED INCIDENTS DO NOT MOVE. Measured as INERT: deleting the count threshold entirely gives the identical result to its best setting, and every other setting only adds false pages.
Ask a model instead	Fixes the half no rule can reach: 6 of 6 silence windows and 5 of 8 quiet-killers, which no threshold or regex managed at any setting.	⚠︎ 25 FALSE PAGES, 24 of them on one broken health check — and it MISSED 2 of 6 cascade windows, the loudest real outage in the corpus. It introduced a silent failure the rules did not have.
Let the model page automatically, on every window	This is what a naive deployment does, and it is the run above.	One unnecessary 3am call every 4 windows of stream. There is no rate limiter here and nothing dedupes across windows, so a cascade pages once per window it spans.
Rules where there is evidence, model where there is none	The rules keep every window with lines to grip; only the windows a threshold and a regex cannot touch are escalated.	⚑ 3 missed and 0 false pages — better than BOTH parents on BOTH numbers — at 41 calls instead of 123. Measured for $0.00 by re-scoring the recorded verdicts, not by a second run.

⚑ READ THE THIRD AND FOURTH ROWS TOGETHER. The exposure is real and structural, and the rate is unmeasured — which is a worse position than a measured bad rate and is published as such. What limits the blast radius is that the output vocabulary is two words: the worst an injection can do is flip one window's verdict, not exfiltrate anything, because there is nowhere for text to go. That is an argument about shape, not a measurement.

The result⚑ THE ATTACK SURFACE HERE IS THE LOG STREAM ITSELF, AND THAT IS NOT A HYPOTHETICAL. Every other kit in this series is handed a document by somebody who at least intended to be a user. This one reads whatever a service wrote to stdout — including whatever an attacker persuaded a service to write. A log line is an untrusted string that arrives in the prompt verbatim, and anybody who can cause one can put text in front of the model.

6027decisions, every setting × every window

0 of 49settings handling all six traps

$0.00cost — no model is called

49 settings across four floors, every one of them scored over all 123 judged windows through the identical src/decide.py the model run uses. The result is the same wherever you stand: NO setting handles all six traps, and the one none of them reaches is quiet-killer — one WARN line about a certificate, which no count reacts to and no regex matches. Raising the threshold does not trade that away; it trades away nothing at all, because the count is measured as INERT at its best setting. ⚑ THE MODEL CLOSES MOST OF THAT GAP: 5 of 8 quiet-killers caught — and opens a new one, 25 false pages the rules did not have.

The output vocabulary is the containment, and it was verified

Every one of the 123 replies was exactly PAGE or HOLD. The parser accepts nothing else and returns None for anything it cannot resolve, which is counted as no_verdict and excluded from both rates — never defaulted to HOLD, which would convert a failed or manipulated call into a calm-looking score.

HonestyWhat this does not prove

The injection rate. Not run, not authorised, and deliberately not estimated.
Whether a longer or adversarially-crafted log line could exhaust the context window and push the real evidence out of it. The corpus has no such line.
Whether the silence check can be attacked by causing a service to log noise it would not otherwise write. Nothing measured it.
Anything about a multi-user deployment. This is a local single-user tool and every claim above assumes that.

Home › Use Cases › Decide when an online store's server logs should wake someone › Guardrails

Use caseUC0012

🧪 Step 11

Guardrails

Decide when an online store's server logs should wake someone

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

the guardrail, verbatim

⚑ THE GATE AND THE THRESHOLD ARE BOTH GUARDS AND NEITHER IS THE MODEL'S. Nothing reaches a model unless src/window.py::candidates passes the window, and nothing wakes anybody unless the decider says PAGE. Both are pure code, both are yours, and the kit publishes what each one costs rather than recommending a setting.

src/window.py — candidates() and gone_silent(); src/rules.py — the threshold and the keyword pattern; src/decide.py — outcome().

EvidenceDoes it hold?

What	Measured
Nothing reaches a model unless the gate passes the window	123 of 240 windows passed; gate recall 1.0 — every one of the 20 windows that should page survived it, measured in src/window.py::stats and published beside every score.
The five outcomes reconcile against the window count	15 + 5 + 25 + 78 + 0 = 123 on r014, asserted by src/decide.py::tally on every run and printed as an equation under the app's tiles.
`no_verdict` is counted apart and excluded from both rates	0 on this run — every one of the 123 replies was exactly PAGE or HOLD and `finish_reason` was `stop` on all of them.
A run whose failed windows outnumber its answered ones cannot be published	Red-proven in build/measured/runlog.py before this run existed: refused at 1-of-123 and at 61-of-123, records at the exact tie.
The panel's threshold moves the free rules and never the model's recorded verdict	tools/shoot_ui.mjs reads the tile numbers out of the DOM at three settings and fails the shot if they did not move — and re-asserts from the UI that no free setting handles all six traps.

The limitWhat a guardrail is not

It is not a content filter. Nothing here inspects the model's words for anything except the two verdicts.
It is not a confidence score. The model returns a word, not a probability, and this kit deliberately does not collect a number it would have to caveat away.
It is not a rate limiter on paging. If the model says PAGE on forty consecutive windows, forty pages go out — which is exactly the pager-storm failure a real deployment must handle and this kit does not.
It is not a third verdict. There is no UNSURE, because a pager is a binary actuator and mapping hesitation onto 'hold' would hide missed incidents inside a calm score.

WatchedWhat is watched, and why that one

1run recorded

0 / 0deterministic metrics exact

+0.0%largest move — latency

0band breaches

—model half · needs provider URL

The kit measures 21 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

11 measured by the latest run 10 need the model half

Metric	Owner	Role	Why this one
verdict-against-label	The model's word against the window's label	alarm	missed_incident; false_page; no_verdict; gate_recall — alarm on Any missed_incident on a cascade window — the loudest and least ambiguous class in the set. This run had 2, which is the finding that should worry a reader most.

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

Guard	Value	If it moves
corpus.doc_count	240	different corpus — nothing is comparable
corpus.bytes	970,580	windows of logs edited — the count held, the bytes did not
split.count	240	the windows count moved — a different set was scored
split.size_p50	89	the median size of one window moved
split.size_p95	105	the 95th-percentile size of one window moved
dataset.rows	240	the test set changed — every rate has a new denominator
tokens.context_limit	1,000,000	different model family
index.build_seconds	2.0	no index is built here — a change is in corpus preparation, not an index rebuild

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

Group	Band	Denominator	Where the band comes from
triage quality	not yet known	123 windows, of which 123 answered	A band is the spread between runs of the same set, and there has been one scoring run. A second tier on the identical 123 windows is 123 calls and is not authorised, so this stays 'not yet known' rather than being filled with a single point wearing a range.
failure taxonomy	not yet known	123 windows, and the five reconcile against it	⚠︎ THE TWO EXPENSIVE ONES MOVE IN OPPOSITE DIRECTIONS AND MUST NEVER BE SUMMED. r014 recorded 5 missed incidents and 25 false pages; a band drawn over their total would hide a system that stopped waking anybody.
reliability	0 of 123 on the one scoring run	123 windows	Every reply was exactly PAGE or HOLD and `finish_reason` was `stop` on all 123 calls. ⚠︎ THIS IS THE ONE TO WATCH FIRST ON ANY RE-RUN: a silent model pages nobody, so a broken run renders as a calm night with zero false pages.
latency	not yet known	123 calls on one tier	p50 3544 ms, p95 15999 ms — a spread of 4.5×, because this model reasons before answering and the windows it reasons longest about are the ambiguous ones.
cost	not yet known	123 windows on one tier	157,449 in / 56,842 out for $0.037959. ⚑ The output side is the unstable half: the reply is one word and the average 462 output tokens are almost all reasoning.
the free floor	measured, and it is not a band either	123 windows at each of 49 settings across four floors	⚑ EVERY SETTING WAS SCORED, SO THIS IS THE WHOLE CURVE RATHER THAN A SPREAD — and its shape is the finding: the error-count threshold is INERT at its best setting, proved with a no-count control row.

HistoryRun history

1 recorded run. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked.

Metric	r014-ops-triage-flash 2026-08-13
input tokens, whole run	157449
model latency p50 ms	3544.01
model latency p95 ms	15999.04
output tokens, whole run	56842
triage detection	0.75
triage page precision	0.375
false page	25
held correct	78
missed incident	5
no verdict	0
paged correct	15

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 11 chips that all say so.

DeviationsWhat deviated

0 breaches across 1 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

Lever	What moves	Basis	Evidence
Change the bucket size	every label · every score · the meaning of every window	measured	⚠︎ NOT A SETTING, A DIFFERENT QUESTION. Widen it and the retry storm's recovery line falls inside the window, so the correct answer flips from page to hold; narrow it and one cascade becomes four windows that each look like a separate outage. ⚑ MEASURED ON THE FLOOR FOR 0 CALLS (evals/recut.py): ten minutes costs 24.7 points of detection and fifteen costs 24.3, page precision unmoved at 100%, and the absence rule stops firing altogether — 6 silent windows found at five minutes, 1 at ten, 0 at fifteen. ⚠︎ The MODEL half is NOT measured, because r014's verdicts were reached on five-minute windows; `window_seconds` rides as a comparability guard on every run so two cuts can never be read as one series.
Raise the count threshold	false pages ↓ · missed incidents unchanged	measured	20 → 49 on the free floor: false pages 58 → 0, missed incidents 14 → 14. And deleting the threshold entirely gives the identical result to 49, which is how it was shown to be inert rather than merely unhelpful.
Loosen the gate	cost ↑ · gate recall cannot fall	measured	123 of 240 windows currently pass. Tightening it is the dangerous direction: a window the gate drops is held by default and no model quality recovers it — the first version of the absence check scored gate recall 0.85 for exactly that reason.
Change the keyword pattern	the FLOOR moves · the model does not	measured	The pattern is what the model is compared against, so editing it moves the published comparison without touching the model at all. It currently catches every cascade for nothing and cannot match ordinary English about a certificate.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

Band	What fires
triage quality	nothing yet
failure taxonomy	nothing yet — one run
reliability	any no_verdict at all
latency	nothing yet
cost	nothing yet
the free floor	8 missed / 0 false pages at its best setting

NextThe three you would add first

A pager-storm limiterNothing here stops forty consecutive PAGE verdicts becoming forty pages, and the cascade trap is exactly the shape that produces them. It is the most likely way a real deployment of this kit hurts somebody.
Deduplication across windowsAn incident spanning two windows pages twice. The corpus's incidents span at most two, which is kinder than reality.
A third destination — a morning ticket queueThere are two verdicts because a pager is binary, but 'not now, and somebody should look' is a real answer with nowhere to go. Adding it is a product decision and a different flow, not a word in the prompt.
The escalation rule, in the app rather than only in the reportIt is the design that beat both parents on both numbers at a third of the calls, and right now it exists as a measurement in evals/combine.py rather than as something the kit runs.

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-run python3 -m evals.baseline on any change to src/rules.py or src/window.py, and python3 -m evals.combine on any change to either — both are free, and both are the things that move the number the model is compared against.

What this cannot tell you

Whether gate recall stays at 1.0 on a real stream. It is a property of this corpus and this gate together, and the corpus is invented.
Whether the silence check's 30-minute horizon is the right one. It was chosen to be longer than a pause and shorter than a pager cycle, and nothing measured it.
What the pager-storm failure would cost, because the corpus never produces one longer than two windows.

Home › Use Cases › Decide when an online store's server logs should wake someone › Frameworks

Use caseUC0012

🧪 Step 12

Frameworks

Decide when an online store's server logs should wake someone

In one lineNo framework, deliberately

This kit has no framework and no dependencies at all — csv, re, urllib and http.server from the standard library, and requirements.txt is an empty file with a comment explaining that the emptiness is load-bearing.

The mappingSeam by seam, who owns what

Seam	File here	Framework equivalent	Note
the unit of work	`src/window.py`	stream processors, windowing engines (Flink, Kafka Streams, Beam)	the one station no other kit in this series needs, because every other unit of work arrives already formed — and at this scale a fixed bucket is arithmetic on a timestamp, not a runtime
the pre-filter	`src/window.py::candidates`	alert routers, sampling and guard layers	what a framework would call routing. Its recall is a published ceiling on everything downstream, which is the part a framework will not measure for you
the model call	`src/adapters/__init__.py`	LLM client wrappers, provider routers	the wire format for one completion is small enough that stdlib is honestly the right size, and it keeps the fork test to one install
the evaluator	`src/decide.py`	eval harnesses, scoring libraries	five outcomes and two rates, shared by the model path and the free floor so the two cannot be measured with different instruments

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph. The pipeline has one path — cut, collapse, gate, judge, score — with no branching, no retries beyond the adapter's transient handling and no state carried between windows except the six-window history the silence check reads. A framework would add a scheduler to a for-loop.

The other sideWhat a framework costs you

No dependency resolution, no lockfile, no framework upgrade path — and no framework features either. Anything this kit does not do, a forker writes.
The adapter speaks two wire formats by hand. A third provider is a function, not a config line.
There is no orchestration, so there is no retry-the-whole-window, no checkpointing and no resumption. A run that dies at window 90 starts again.
No tracing. finish_reason, reasoning_chars and the latency are recorded per row and that is the whole of the observability.

What we could NOT verify

Whether a framework would help at production scale. This kit judges 123 windows in one process and never needed a queue; a real stream is continuous and that is a different program.
Whether the escalation architecture stays this cheap when the rules are a real alerting stack rather than 40 lines of re.

Home › Use Cases › Decide when an online store's server logs should wake someone › Observability

Use caseUC0012

🧪 Step 13

Observability

Decide when an online store's server logs should wake someone

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r014-ops-triage-flash on the fast tier, 2026-08-13. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

Reading	Last run	Band	What fires
Model, median	3,544 ms	not yet known	nothing yet
Model, p95	15,999 ms	not yet known	nothing yet
Input tokens	157,449	not yet known	nothing yet
Output tokens	56,842	not yet known	nothing yet

No movement column. This is the only run on record, so there is nothing to move against. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

The other sideWhat this does not tell you

What we could NOT verify

No stage-by-stage span tree. These are whole-call timings; which part of the pipeline spent the time is not recorded by this kit's harness.
Latency is a run-level median and p95. The spread between the fastest and slowest single call is not stored, so the distribution cannot be drawn.
No earlier run is comparable to this one (a different model tier or a changed guard), so nothing on this page is a trend yet.
One machine, one network, one region. Latency includes the provider round trip from a single location and will not transfer to yours.

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

Home › Use Cases › Decide when an online store's server logs should wake someone › Run it in your environment

Use caseUC0012

🧪 Step 14

Run it in your environment

Decide when an online store's server logs should wake someone

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — windows are cut from the log stream per run.

The machineWhat this needs

Dependencies	nothing beyond the standard library — `requirements.txt` names no package
Configuration	7 environment variables — the full knob table is at the bottom of this page
Last verified run	2026-08-13, across 1 committed record

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

Artifact	Where	Leaves your machine?
event stream	`data/events.csv` — 15,189 generated events, fixed seed, no clock read; your disk	never whole; one five-minute window at a time, lines collapsed, per call
windows	cut per run by `src/window.py` — nothing persists between runs	only the ones the gate passes to a model; the rest are held for free
labels	`data/labelled.jsonl` — 240 windows, each carrying its trap kind	never — PAGE/HOLD is scored in-process by `src/decide.py`, no judge model
the key	`.env` — never committed	only inside the Authorization header

The wireWhat crosses it, and when

Runtime call site	Destination
`src/adapters/__init__.py` line 55	the configured BASE_URL

Build-time (corpus tooling, run once)	Destination
none — this kit’s corpus is generated locally

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The third state

What the provider retains, trains on or logs once a request arrives is not measured here — it is provider-dependent, and this page does not assert what no run of ours can verify.

The shapeHow it ships, and what you add

The key	Read from the shared .env or the real environment only, never written into the repo, never logged, never returned by any endpoint. /api/status reports whether a key EXISTS as a boolean and nothing more. ⚠︎ AND THE SHARED .env AT THE REPO ROOT MEANS A BRAND-NEW KIT ALREADY HAS A LIVE KEY — 'I have not configured anything yet' is not the same as 'nothing can spend'.
You add	your reverse proxy and your identity in front of one process — the kit deliberately ships neither, because identity belongs to your estate, not to a kit
Never ships	Docker, Helm, a queue, a database. The product is a folder of readable Python; what to run it under is this page’s job to state, not the kit’s job to impose

The ladderEach decision, and what you provision past its ceiling

Decision	The kit ships	Measured here	Past the ceiling	What stops being valid
windowing	fixed 300-second buckets (`src/window.py`); the gate then passes 123 of 240 windows to a model and holds the rest for free	re-cut at ten minutes and the free floor's detection falls 24.7 points, at fifteen 24.3 — and gone_silent fires on 6 windows at five minutes, 1 at ten, zero at fifteen (evals/recut.py, 0 calls; the model half is unmeasured — r014's verdicts are five-minute verdicts, and window_seconds rides as a comparability guard)	not a setting, the question itself — widen it and a retry storm's recovery line falls inside the window, so the correct answer flips from page to hold; the labels are true for a five-minute cut and no other, so a new width means re-labelling	every published score, and the absence signal outright — a service quiet for five minutes still logs in the other half of a bigger bucket, so the trap this kit was built around becomes undetectable
model	one HTTP completion call per gated window behind `src/adapters/__init__.py` — any OpenAI-compatible endpoint or Anthropic; one word back: PAGE or HOLD	—	a hosted provider or a local server — the .env decides, not the code. The payload here is your production logs, including whatever an attacker persuaded a service to write, so the zero-egress path is worth more on this kit than on its siblings	published detection and precision are per-model — the floor, the gate and the scorer do not move, so the comparison re-runs on yours
labels	`data/labelled.jsonl` — 240 windows labelled by the generator that planted the incidents; `evals/check_labels.py` re-derives every label by a different code path and refuses the set on disagreement	123 of 240 judged after the gate; gate recall 1.0 — every one of the 20 windows that should page survived it (lenses.Eval.dataset and guardrails.holds, r014-ops-triage-flash)	label windows from your own stream — and expect the quiet-service list to grow: `deploy` is excluded from the silence check by name because silence is its health, and on a real estate that list is long and nobody maintains it	the labels are a property of the generator AND the cut — your stream has no planted truth, so the labelled set is yours to build before any score means anything

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

What you see	What it is	First move
the same service paged again and again for a line that has never once meant anything	a flapping health check — 24 of the run's 25 false pages are one broken check the model kept reacting to	fix or fence the check before touching the prompt — and remember nothing here rate-limits paging, so forty PAGE verdicts are forty pages (lenses.Eval.taxonomy (false_page), r014-ops-triage-flash)
the loudest real outage in the stream — postgres FATAL, five services down — held	the model missed 2 of the 6 cascade windows that a plain count threshold catches for nothing	run the free floor beside it (`src/rules.py`) and page on either — the rules currently catch every cascade for $0.00 (lenses.Eval.taxonomy (missed_incident) and lenses.Business.not_good_enough, r014; guardrails.ripple for the floor's cascade record)
No machine symptom — this failure leaves no trace in any output.		a service that dies in the stream's opening half-hour produces no symptom at all — `gone_silent` refuses to answer without six windows of history rather than guess, so the first 30 minutes of any stream are judged without the absence signal (lenses.Data.breaks_on). The control is knowing that window exists and not trusting early-stream holds

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

Constant	File	Value
`FLOOR_THRESHOLD`	`evals/combine.py`	49
`BASE_S`	`evals/recut.py`	300
`HISTORY_MINUTES`	`evals/recut.py`	30
`RETRIES`	`src/adapters/__init__.py`	4
`MAX_TOKENS`	`src/prompt.py`	4096
`DEFAULT_THRESHOLD`	`src/rules.py`	20
`HISTORY`	`src/window.py`	6
`WINDOW_S`	`src/window.py`	300

The other sideWhat this page does not claim

Not measured, and said so

Concurrency and GPU sizing — no run produced them. Provider-side retention and log residency — provider-dependent, and on this kit the payload is production logs, so that unknown IS the posture question. Where a window stops fitting in context — this corpus's noise collapses well (a median 89 raw lines became 52 distinct); a real stream with high-cardinality messages collapses far less, and nothing measures where the ceiling is. The model half of the re-cut finding — only the floor was re-cut, for 0 calls. And what a pager storm or a long incident costs: the corpus never produces one longer than two windows, there is no cross-window deduplication, and a run that dies at window 90 starts again — no checkpointing, no resumption.

The corpus licence, from the Data lens: MIT — this repository's own licence, verified against LICENSE at the repo root on 2026-08-13. Nothing is fetched and no third-party grant is involved, because every event is invented here. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

Home › Use Cases › Decide when an online store's server logs should wake someone › Evals › The model's word against the window's label

Use caseUC0012

🧪 Evals · grader

The model's word against the window's label

Decide when an online store's server logs should wake someone

In one lineThe model's word against the window's label

Did the model's word — PAGE or HOLD — agree with what the corpus knows that window to be?

$0.00per 1,000 windows

nodata leaves your network

yessame answer every time

MethodHow the test was run

src/decide.py::outcome, in-process, no key and no model. The same function evals/baseline.py and evals/combine.py call.

The inputOne real row, seen by every grader

The unit of work

one five-minute window, w0xx

What a rule engine has to go on

47 error/fatal lines, no keyword match, no service silent

The free floor

HOLD at threshold 49 — correct

The model

PAGE

Ground truth

hold — a health check that has failed identically for nine days

Scored as

false_page

Grader	Verdict	Why
The model's word against the window's label	false_page	Labelled `hold` — a health check that has failed identically every six seconds for nine days. The free rules hold it at their best setting; the model returned `PAGE`, so somebody is woken. Counted as a false page, which is the direction that gets a pager ignored. 24 of the run's 25 false pages are this same trap.

The formulaWhat it computes

detection = paged_correct / (paged_correct + missed_incident); page_precision = paged_correct / (paged_correct + false_page). ⚑ BOTH EXCLUDE `no_verdict`, which is why `answered` is published beside them — and on this kit that exclusion matters more than on any sibling, because a model returning nothing pages nobody and therefore looks like a calm night.

The analysisWhat it actually did

Model	Result
the fast tier	75.0% detection · 1 more measured on this row

In operationWhat to monitor

Reference standard: The corpus's own labels, written by tools/build_corpus.py at the moment each window was generated and re-derived independently by evals/check_labels.py. ⚑ THIS GRADER IS THE REFERENCE, so its own TPR and TNR are UNKNOWN by definition — it cannot be scored against itself, and a number there would be circular.

These rates are UNKNOWN, on purpose

This grader's own error rate is not measured and largely cannot be — it IS the reference. What can go wrong is the LABELS, and those come from the generator, which is why evals/check_labels.py re-derives them by a different code path.

Watch these

missed_incident
false_page
no_verdict
gate_recall

Alarm on

Any missed_incident on a cascade window — the loudest and least ambiguous class in the set. This run had 2, which is the finding that should worry a reader most.

How tight can the band be? There is no tuned threshold on the MODEL path: its word decides. The threshold belongs to the free floor, and it is swept over 49 settings rather than chosen — with a no-count control row proving it is inert at its best setting.

Cadence: Re-run on any change to src/prompt.py, src/decide.py or the corpus generator. The free half (evals/baseline.py, evals/combine.py) costs nothing and should be re-run on every change to src/rules.py or src/window.py.

The decisionWhen to reach for it

Use it

The truth is known and the answer is a word from a closed list of two.

Do not use it

The truth is not known — which is the normal state of a real log stream, and the reason this corpus is generated rather than captured.

A living map of modern AI — kept current every morning

Today's manual process, and the same job with the app

✕Today's manual process

✓With the app

One real case: what the app reads, step by step

How it is built, and how we measured it

What you see is a reference app. We shape it to how you work.

Talk to us

The business caseThe problem this solves

Audience

The inputThe actual windows of logs

The corpus

The outcomeWhat a good result looks like

And when it cannot

Where it fitsWhat did work

At a glanceHow the whole thing runs

Should you use this?What you bring, where it stops, and when not to use it

Run itHow this reaches your data

Run this on your own data

Step 01 of 14Business case

In brief

Current processWhat it replaces

Where it is not good enough

Step 02 of 14Architecture

In brief

The swap seams

Components

Where it breaks at scale

Step 03 of 14App UI

SuccessesWhen it works

LimitsWhen it does not

Step 04 of 14Corpus

In brief

How it is cutWhat one window is

SetupWhat the setup figure measured

LicenceLicence

Bring your ownBring your own windows of logs

What breaks it

Step 05 of 14Prompt

In brief

AssemblyHow the prompt is assembled

Full promptThe prompt in full

Raw responseThe raw response

Step 06 of 14Evals

In brief

The rulerHow it was graded

MeasurementsWhat was measured

How the method was validated

Grading costWhat it costs

Same work, 8× the bill

Grading unitWhat the grading figure prices

The gradersOne way to grade, and why it is the only one

Why this set cannot separate them

Choosing oneWhen to reach for the model at all

Failure causesHow it fails

What we could NOT verify

Step 07 of 14Unit cost

In brief

Answering vs gradingGrading is free here, and it is a measured zero

Cost driversWhat actually moves the bill

Your volumeWhat it costs at your volume

Where pricing changes shape

Your return, with your numbers

Model choiceWhy this model

Other modelsOn other models

Read this against the numbers above

The codeEvery file, and what it is for

tools/build_corpus.pycorpus generator

src/window.pythe window cutter and the gate — a swap seam

src/rules.pythe rule floor — a swap seam

src/prompt.pythe prompt

src/adapters/__init__.pythe model adapter — a swap seam

src/decide.pythe decider — a swap seam

evals/baseline.pythe free floor harness

evals/combine.pythe combination harness

src/app.pythe app

Start hereThe shortest path into it

ScenariosWhat it costs at your volume

Grading really is free here, and that is a property of the ruler rather than a discount

OperationsFour shapes of workload

What this ignoresWhere a real bill diverges

src/adapters/init.pythe model adapter — a swap seam