The business caseThe problem this solves

A bundle of alerts that look related is two questions, not one, and getting either wrong is expensive in a different direction. Call a genuine phishing report a false positive because it is worded as a calm 'is this legit?' question, and an incident goes unworked. Merge two unrelated alerts into one case because they share a source IP -- the same building's or VPN concentrator's egress address -- and the real second incident is hidden behind the first one's containment action while the analyst's read is spent on a coincidence. A SOC analyst reading a bundle of 2-4 correlated-looking alerts, deciding which are real and which are explained away, deciding which of the real ones are actually the same incident rather than a coincidence, and drafting the containment note -- for every bundle, every shift.

The inputThe actual case windows

The corpus is 33 case windows, 0.03 MB (jsonl 1). A real SOC's alert queue names real users, real hosts and real attacker infrastructure -- this repo has never held one and is not able to. There is no public corpus of paired (alert group, adjudicated disposition, confirmed incident grouping) records, for the same reason no public corpus of production payment runs or vendor contracts exists: the interesting cases are exactly the ones nobody can publish. Every 'external' IP in this corpus is drawn from the RFC 5737 documentation ranges (203.0.113.0/24, 198.51.100.0/24), which cannot resolve to anyone's real host; every user, host, domain and hash is invented.

{"id": "cw001", "window_start": "2026-08-17T06:37:00Z", "on_call_analyst": "Priya Shah", "alerts": [{"alert_id": "ALT-0001", "alert_type": "malware_detection", "description": "Endpoint agent flagged an unsigned binary establishing a reverse shell connection shortly after execution.", "entity": "host:crm-db-05", "indicators": {"file_hash": "d1dc5e74f852a7c2750a5ea08317e789d25ecb0b", "process_name": "sysmaint.exe", "host_ip": "10.20.174"}, "ts": "2026-08-17T06:37:00Z"}, {"alert_id": "ALT-0002", "alert_type": "data_exfil_alert", "description": "Unusual outbound transfer of a large archive to an external IP shortly after the malware alert on the same host.", "entity": "host:crm-db-05", "indicators": {"dest_ip": "203.0.113.106", "bytes_transferred": "1148737880", "source_host": "crm-db-05"}, "ts": "2026-08-17T06:38:00Z"}]}
{"id": "cw002", "window_start": "2026-08-17T07:14:00Z", "on_call_analyst": "Marcus Webb", "alerts": [{"alert_id": "ALT-0003", "alert_type": "malware_detection", "description": "Endpoint agent flagged an unsigned binary establishing a reverse shell connection shortly after execution.", "entity": "host:db-12", "indicators": {"file_hash": "705b10bbfc5641aee6bf795c8840e6f0ab0d4179", "process_name": "sysmaint.exe", "host_ip": "10.20.72"}, "ts": "2026-08-17T07:14:00Z"}, {"alert_id": "ALT-0004", "alert_type": "data_exfil_alert", "description": "Unusual outbound transfer of a large archive to an external IP shortly after the malware alert on the same host.", "entity": "host:db-12", "indicators": {"dest_ip": "203.0.113.221", "bytes_transferred": "3886850958", "source_host": "db-12"}, "ts": "2026-08-17T07:19:00Z"}]}

The outcomeWhat a good result looks like

A true_positive/false_positive disposition for every alert in the window, the true positives grouped into the incidents they actually belong to, and one drafted containment recommendation per incident citing a real indicator -- for the named on-call analyst to approve before anything is done.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Deciding which alerts in a pre-bundled candidate group are real and which are explained away — the fast tier, over the free keyword floor
  100%% disposition accuracy and 0 of 52 missed true positives this run, against the floor's 69.5%% and 4 of 52 -- including 2 of the 3 calmly-worded phishing reports a keyword scan has nothing to catch.
• Deciding which true positives are genuinely the SAME incident — the fast tier's case grouping, read pair by pair rather than group by group
  0 of 47 gold-different pairs merged this run, including all 8 planted coincidental-indicator pairs, against a floor that merges 8 of 8 by construction. Pair-wise is the honest unit: a partial merge inside a bigger group is still a false correlation for every pair it touches.
• What this kit is not — a disposition, a grouping and a drafted recommendation for a named on-call analyst to approve -- never a containment action
  There is no function in src/triage.py, src/app.py or the UI that locks an account, blocks mail flow or isolates an endpoint. Every recommendation is text.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-20. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-20 — r001-it-sectriage. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — A fresh clone with no key configured renders the whole panel -- every window's alerts, indicators and gold grouping, since data/windows.jsonl and data/gold.jsonl are committed, not fetched (python -m src.app). Clicking 'Ask the model' with no API_KEY returns a calm 200 explaining nothing was called. It cannot reproduce a disposition, a score or a dollar figure without a key -- those are what results/eval-r001-it-sectriage.json already committed. The free floor (python -m evals.baseline) and the corpus self-check (python tools/build_corpus.py --verify) both run on a cold clone with no key at all.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. END-TO-END per case window: one HTTP request carrying every alert in the window with its type, entity, timestamp, description and indicators, parsed into a disposition per alert, a case grouping and a drafted recommendation per case. No retrieval step -- loading a window from disk is pure code and costs no network time. Reasoning was left at the provider's default (on), so p50 and p95 both include a reasoning pass on every call; the p95 of 47,346 ms IS one specific window, cw026, the corpus's hardest correlation trap, which spent 22,431 reasoning characters and 5,442 output tokens on it. The spread is real and it is the workload's, not the network's: the fastest call took 1,644 ms.

Current processWhat it replaces

A SOC analyst reading a bundle of 2-4 correlated-looking alerts, deciding which are real and which are explained away, deciding which of the real ones are actually the same incident rather than a coincidence, and drafting the containment note -- for every bundle, every shift.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureEvalsEvalsEvalsEvalsPromptPromptPromptPromptArchitectureArchitectureApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• jsonl
• 2 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one characters per window's assembled alert block is

One case window, every alert in it included whole -- no train/test split, every window judged once, in one call. 18 windows carry 2 alerts, 14 carry 3, 1 carries 4.

SetupWhat the setup figure measured

No index is built. src/app.py finds a window by scanning T.windows() for a matching id -- 33 rows, a linear scan, not a search structure -- and src/triage.py sends that window's alerts whole into the one call. build_seconds and build_cost_usd are both zero because there is no index-build step, not because one ran for free.

LicenceLicence

MIT (this repository's own licence)

Bring your ownBring your own case windows

Point tools/build_corpus.py's window plan, alert templates and indicator vocabulary at your own alert types, entities and case shapes, or write your own data/windows.jsonl in the same shape -- src/triage.py and src/prompt.py read a window by its own fields and do not care where they came from. The two-value disposition vocabulary in src/prompt.py assumes exactly true_positive and false_positive; a third value (a 'needs enrichment' state, say) is a prompt.py AND tools/build_corpus.py change, not a data-only change.

⚠︎ And what stops being true when you do: The measured 100%% disposition accuracy and 0%% missed_true_positive / false_correlation rates are THIS corpus's alert phrasing (a fixed set of description templates per alert type) and THIS corpus's trap design (3 mundane phishing reports of 9 true-positive phishing alerts; 8 coincidental-indicator windows of the 26 that present a genuine merge-or-don't decision). A real SOC's own alert text, and a real network's own reasons for two users to share an egress address, are both unmeasured by this run.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — but they are the input total apportioned by each part’s share of the characters, not a per-part reading from the provider’s counter. The total is measured; the split is arithmetic, and a part carrying denser text really costs more of it than this says.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 967 tokens assembled, 805 are instructions — 83% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Logged verbatim by importing src.prompt and src.triage directly and calling prompt.build(window) for cw001 (not retyped) -- the literal system and user content src/triage.py::check() sends. Per-part token counts are a proportional estimate over character share (system 2778 / alerts 560 chars, 3338 total) applied to this call's own total of 967 input tokens -- the provider reports only the call's total, matching lenses.LLM.tokens.input exactly, never a per-segment split. The 'alerts' part covers only the alert block src/prompt.py::_alert_block() builds; the surrounding 'Case window ... Return the JSON object ...' instruction text is part of the user message actually sent (see prompt_verbatim) but is not a separately named segment in the kit's own code.

Full promptThe prompt in full

You are a SOC triage analyst. You are shown one case window: a small group of 2-4 security alerts that were bundled together because they occurred close in time and/or share an entity. Your job has three parts, in one reply:

1. DISPOSITION -- for every alert in the window, decide true_positive (a real security event) or false_positive (benign, correctly explained away). Judge each alert on its own evidence.
   - A calm or routine-sounding report is not evidence of a false positive. Genuine phishing reports are often written as an ordinary 'is this legit?' question, not an alarm -- judge the sender domain, the link and the ask, not the tone.
   - A loud description is not automatically a true positive either -- read whether it is actually explained (a confirmed backup window, a confirmed travelling user, an approved software request).

2. CORRELATION -- among the alerts you called true_positive, decide which ones are genuinely the SAME incident and which are separate. Two alerts belong in the same case only when the evidence connects them causally, not just superficially.
   - Do not merge two alerts into one incident merely because they share an entity, a close timestamp, or one matching indicator value. A shared source IP can mean one attacker, or it can mean two unrelated users behind the same building's network or the same VPN concentrator -- read what the shared value actually is before treating it as a link.
   - A true positive with no genuine partner is its own case, alone.
   - Never place a false_positive alert into any case -- a case exists only where there is a real incident.

3. RECOMMENDATION -- for every case with at least one true-positive alert, draft one short containment recommendation (account lockout, mail-flow block, or endpoint isolation, whichever fits). Cite the specific indicator(s) that justify it, in the exact form they appear in the alert data (name=value). Never cite an indicator that is not actually present on an alert in that case, and never invent one.

YOU NEVER EXECUTE CONTAINMENT. You only draft the recommendation. The named on-call analyst shown to you reviews and approves it before anything is done -- your recommendation text should read as a draft awaiting that approval, not as an action taken.

Reply with ONLY a JSON object, no other text, with exactly these three keys:
  "alert_dispositions": {"<alert_id>": "true_positive"|"false_positive", ... one entry per alert in the window}
  "case_groups": [[<alert_id>, ...], ...]  -- one list per incident, true-positive alert ids only, every true positive in exactly one group
  "recommendations": [{"case": [<alert_id>, ...], "action": "<one or two sentences>", "citations": ["<indicator_name>=<value>", ...]}, ...]  -- one entry per case_groups group

Case window cw001, starting 2026-08-17T06:37:00Z. On-call analyst: Priya Shah.

Alerts in this window (ALT-0001, ALT-0002):
  ALT-0001  [malware_detection]  entity=host:crm-db-05  06:37:00
    Endpoint agent flagged an unsigned binary establishing a reverse shell connection shortly after execution.
    indicators: file_hash=d1dc5e74f852a7c2750a5ea08317e789d25ecb0b, process_name=sysmaint.exe, host_ip=10.20.174
  ALT-0002  [data_exfil_alert]  entity=host:crm-db-05  06:38:00
    Unusual outbound transfer of a large archive to an external IP shortly after the malware alert on the same host.
    indicators: dest_ip=203.0.113.106, bytes_transferred=1148737880, source_host=crm-db-05

Return the JSON object described in the system message. Every one of these alert ids must appear in alert_dispositions, exactly once: ALT-0001, ALT-0002

Raw responseThe raw response

{
  "alert_dispositions": {
    "ALT-0001": "true_positive",
    "ALT-0002": "true_positive"
  },
  "case_groups": [
    [
      "ALT-0001",
      "ALT-0002"
    ]
  ],
  "recommendations": [
    {
      "case": [
        "ALT-0001",
        "ALT-0002"
      ],
      "action": "Isolate host crm-db-05 immediately and block outbound traffic to the external destination to contain the reverse shell and stop further data exfiltration. Preserve the endpoint for forensic collection.",
      "citations": [
        "file_hash=d1dc5e74f852a7c2750a5ea08317e789d25ecb0b",
        "process_name=sysmaint.exe",
        "dest_ip=203.0.113.106",
        "source_host=crm-db-05"
      ]
    }
  ]
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Sort a security team's alert bundles into real incidents — 33 case windows. One model answered, and every answer was then graded one way — by == against a fixed gold value, which is a measurement too and only as good as the gold behind it.

The rulerHow it was graded

The grader is evals/scoring.py and it is pure code: exact match on each alert's disposition, a pair-wise comparison for correlation, and an exact (indicator_name, value) lookup for every citation. No LLM judge anywhere in the path, and no retrieval step to score.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran r001 publishes no rate card this repo commits, so nothing here is what was actually paid -- the real spend for this kit's build is recorded in the commit history and the shared call ledger, not on this page. Reasoning was left at provider default (on) with no way to disable it, so the projected figure prices a reasoning pass that a reader on another provider may or may not be billed for.

Grading unitWhat the grading figure prices

Grading calls nothing. Disposition accuracy, missed_true_positive, false_correlation and citation_validity are all pure code over the committed run file, so a forker re-scores this run -- or the free floor -- for $0.00 and needs no key.

The gradersOne way to grade, and why it is the only one

The floor is correct on every window that carries no trap, and it fails exactly where the corpus means it to. It over-merges 8 of 8 planted coincidental-indicator pairs -- 100%% -- because a rule that groups first by raw similarity and only then asks whether the group looks bad will drag an unrelated alert into a real incident's case the moment it shares one indicator value. It misses 2 of 3 mundane-phishing trap alerts (the third is rescued by a loud campaign neighbour it is genuinely correlated with, in cw006). Its 19 of 47 false correlations and 30.5 points of lost disposition accuracy are not a strawman weakening: trusting malware/exfil/brute-force alert TYPES outright is what a rotation under time pressure does, and 'same IP, must be one incident' is what that looks like as code. It also misses a quiet, no-precursor account-takeover login in cw020 and cw024 -- a real gap in the floor that NEITHER named trap set out to test, printed by the baseline rather than smoothed over.

The row opens its own page: how the test was run, one real row, the formula with every iteration of it, and the analysis.

Choosing oneWhen to reach for the model at all

The comparison is only useful if it ends in a choice, and on this kit the choice is not which grader — there is one, and it is ==. It is whether to call a model for this field at all, and the answer is conditional.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The keyword-and-shared-indicator baseline (evals/baseline.py) and the scorer (evals/scoring.py) are both pure code and cost $0.00 to run against either result set. The figure above is this run's own 32892 input / 54550 output tokens priced at Google Gemini 3 Flash's published rate, the same basis cost_per_query_usd uses, not a second, larger spend.

Cost driversWhat actually moves the bill

• The provider's reasoning pass, and it is not a rounding error -- it is the bill. reasoning_chars totals 208,731 across the 33 calls, roughly 52,183 tokens at four characters each, an estimated 95.7% of the 54,550 output tokens. Output tokens are 90.9% of the projected cost, so an estimated 86.9% of this kit's entire dollar figure is reasoning nobody asked for and this kit ships no knob to decline.
• Every alert in the window with its description and 2-4 indicators, plus the fixed system prompt -- essentially flat per call (input tokens ranged 932-1161 across all 33 calls), the floor every call pays regardless of how many alerts the window holds.
• The fixed system prompt (2778 characters -- the three-part instruction, the two pieces of general triage guidance and the output schema) is sent in full on every call: 83.2% of the system+alerts character total on the published call.

Your volumeWhat it costs at your volume

Linear in windows: each call is independent and self-contained, with no shared context or retrieval step to amortise. This run's 33 windows cost about $0.1801 projected onto Google Gemini 3 Flash's published rate, so ten times the set is about $1.801 on the same rate and the same reasoning-on configuration -- arithmetic on the measured per-call rate, not a second run. It is linear in WINDOWS and not in ALERTS: a wider window costs more per call, and how much more was never measured past four alerts.

Model choiceWhy this model

The fast tier was the only model run against this corpus. No second tier was run -- unlike data-reconcile's fast/reasoning comparison -- so this page prices one model, not a trade-off. Note the awkward shape that produces here: a 'fast tier' whose p95 is 47 seconds and whose output-token average is larger than its input, because the provider's default reasoning pass is doing most of the work. See Eval.could_not_verify for what a second run would need to answer.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 33 case windows triaged, 82 alerts dispositioned, 47 gold-different pairs and 109 citations scored by pure code. This run (r001-it-sectriage) answered 82 of 82 dispositions -- the row this table prices, the same one Cost.cost_by_model[0] uses.

The codeEvery file, and what it is for

Eight modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

tools/build_corpus.pycorpus generator — a swap seam

Generates 33 case windows and 82 alerts from a fixed seed (SEED=20260819) across five alert types, in five window patterns (7 correlated_tp, 6 uncorrelated_fp, 7 mixed_tp_fp, 5 multi_incident, 8 false_correlation_trap). Gold is DERIVED, never authored beside the data: every alert is tagged at construction time with a disposition and a case_id, and derive_gold() turns those per-alert facts into the alert_dispositions dict and the case_groups list -- used both to write data/gold.jsonl and, independently, by --verify re-reading both files from disk.

You change it to: Set by the five-pattern window plan, not one global -- a new corpus states its own pattern counts in the same shape, and evals/scoring.py takes the trap alert ids and trap pairs as explicit arguments read from gold rather than re-guessing which windows look like traps.

# Generate the case windows and gold labels this kit checks against, from a fixed seed.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
SEED = 20260819
START = datetime(2026, 8, 17, 6, 0, 0, tzinfo=timezone.utc)
ANALYSTS = ["Priya Shah", "Marcus Webb", "Dana Okafor", "Tomas Rivera"]
USERS = ["jrivera", "asingh", "mchen", "dokafor", "lferreira", "kwalsh", "tnguyen", "rpatel",
HOSTS = ["web-03", "db-12", "fin-app-02", "vpn-gw-01", "mail-relay-04", "build-agent-07",
ALERT_TYPES = ("phishing_report", "suspicious_login", "malware_detection", "data_exfil_alert",
EXT_IP_A, EXT_IP_B = "203.0.113", "198.51.100"

src/prompt.pythe prompt — a swap seam

The three-part instruction (disposition, correlation, recommendation), the output schema and MAX_TOKENS, declared once and read from here by build() and parse(). States both failure modes to the model as ordinary SOC judgement -- a calm report is not evidence of nothing, a shared address is not evidence of one incident -- rather than as hints about this corpus.

You change it to: MAX_TOKENS, and on this workload it is not a formality -- 3000 and 4096 both truncated and both produced a fake missed-true-positive rate. Changing it is a paid re-run, not a re-score; see Business.not_good_enough.

# Assemble the one prompt this kit sends, and parse the one reply it gets back.

DISPOSITIONS = ("true_positive", "false_positive")
MAX_TOKENS = 8192
SYSTEM = (
DEFAULT_PROMPT = "v1"
SYSTEMS = {"v1": SYSTEM}
def _alert_block(a):
def build(window, prompt=DEFAULT_PROMPT):
def _strip_fence(text):
def parse(raw):

src/triage.pythe AI layer

Loads one case window, calls the model ONCE with every alert in it, and parses the reply into dispositions, case groups and recommendations. One call per WINDOW, never per alert -- that is what makes correlation possible at all, and it is the whole difference from ops-triage's per-window judging, which never compares one alert against another. Never executes containment: there is no function here, in src/app.py or in the UI that locks an account, blocks mail flow or isolates an endpoint.

# Triage one case window: classify, correlate, draft -- one model call, done.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
WINDOWS = os.path.join(HERE, "data", "windows.jsonl")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
MAX_TOKENS = P.MAX_TOKENS
def windows():
def load_gold():
def check(cfg, window, complete=None, prompt=P.DEFAULT_PROMPT):

src/adapters/__init__.pythe model adapter — a swap seam

SEAM -- the model. Raw HTTP for every provider (openai-compatible, anthropic), stdlib only. Records finish_reason and reasoning_chars beside the text, which is the only reason the MAX_TOKENS truncation finding was diagnosable without spending again. Takes NO thinking parameter -- see Business.not_good_enough.

You change it to: NOTHING TODAY, AND THAT IS THE FINDING. This kit ships no thinking parameter, so reasoning is whatever the provider defaults to on both the run and the live app. Making it a seam means adding the kwarg rcv-disc's adapter already carries -- an estimated 86.9% of this kit's projected cost sits behind a knob that does not exist yet.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
def complete(cfg, system, user, max_tokens=1024):

src/budget.pyspend control

An append-only ledger written BEFORE each call, shared by every kit under one .env so the daily call cap is on the KEY rather than per kit. Counts calls, not dollars, on purpose.

# A cap on live calls, shared by every kit on this machine. No dependency, no service.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
KIT = os.path.basename(HERE)
ROOT = os.path.dirname(os.path.dirname(HERE))
SHARED_ENV = os.path.join(ROOT, ".env")
LEDGER = os.path.join(ROOT if os.path.exists(SHARED_ENV) else HERE, ".calls-ledger.jsonl")
PAID_ARMS = ("r", "x")
FREE_ARMS = ("t", "b")
BOOT = os.urandom(6).hex()
PID = os.getpid()

src/app.pythe app

Static files and three JSON endpoints (/api/state, /api/window, /api/triage) on the standard library, port 8792. Renders every window, its alerts and the gold answer with no key configured; only /api/triage calls a provider, and with no API_KEY it returns a calm 200 saying so. Reaches the provider down the same adapter with the same defaults the registered run used -- no reasoning-setting discrepancy, unlike rcv-disc and fin-payrun.

# The minimal local UI. Standard library only -- python -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
PORT = int(os.environ.get("PORT", "8792"))
class H(BaseHTTPRequestHandler):
def main():

evals/scoring.pythe scorer

Four things scored, kept apart, never folded into one accuracy number: disposition accuracy, missed_true_positive (the expensive direction for classification), false_correlation (the expensive direction for grouping) and citation_validity. false_correlation is counted over PAIRS rather than groups, because a partial merge inside a bigger group is still a false correlation for every pair it touches and no group-vs-group comparison describes it.

# Score a set of predicted case-window triages against gold. Pure code, shared by

DISPOSITIONS = ("true_positive", "false_positive")
def _gold_group_map(gold_row):
def _pred_group_map(case_groups):
def _norm_kv(s):
def citation_is_real(citation, case_alert_ids, alerts_by_id):
def score(records, gold, windows_by_id, fn_trap_alert_ids, fc_trap_pairs):

evals/baseline.pyfree baseline

A keyword-and-shared-indicator rule written the way a person free-texting a quick triager would write one: trust the three detector types that sound serious, keyword-match the two a human reports, then group anything sharing an entity or ANY raw indicator value and spread the disposition by association. Not weakened for the demonstration -- it is what 'same IP, must be one incident' looks like as code, and it is correct on every window carrying no trap.

# What a simple, dumb rule catches, over the same corpus. Free. No key, no dependency, no model

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
TRUSTED_TYPES = ("malware_detection", "data_exfil_alert", "brute_force")
ALARM_RE = re.compile(
def classify(win):
def main():

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 996 input and 1653 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's alerts are entirely synthetic (tools/build_corpus.py) -- no untrusted party wrote a single description or indicator, and every external IP is from the RFC 5737 documentation ranges. That is a statement about this CORPUS, not about the architecture: in a real deployment, alert descriptions and indicators arrive from detection tools that are themselves reporting attacker-controlled content, which is exactly the shape an indirect prompt injection needs.

Read from the shared .env or the real environment only, never written into the repo, never logged, and scrubbed out of any error string src/app.py returns to the browser.

The experimentWe did not attack it -- and one boundary that should exist does not

An indirect prompt injection needs a field an outside party controls that reaches the prompt verbatim. In a real deployment this kit has several -- every alert description and every indicator VALUE -- and src/prompt.py sends them through unescaped. In THIS corpus every one of them was written by tools/build_corpus.py, so the attack surface exists in the architecture and not in the run. Naming that is the honest version of a resistance rate we do not have. Confirmed by reading the code, not by a run, on 2026-08-20 -- no attack run was fired; see redteam.why.

Each boundary above was checked by reading the call sites, not by an attack trial. Gate 1 is the strong one -- it holds because the function does not exist, which is the only kind of guarantee this kit offers. Gates 2 and 3 are separations and offline checks respectively, and neither is enforced live.

The result0 attack trials, one boundary that DOES hold in code (nothing here can execute containment), and one that does not (nothing cross-checks the model's own citations or groupings before the UI renders them).

This run's corpus is entirely generated -- no alert's description or indicator was written by an outside party -- so there was no live untrusted input to attack with and no attack was attempted. The 0 fabricated citations is a MEASURED result over 109 citations, not an attack result.

HonestyWhat this does not prove

• Whether the prompt-level boundaries survive hostile alert text -- no red-team run exists for this kit.
• Whether a live citation check would ever have caught a real fabrication -- this run's model never produced one to test against.
• Whether a model that grouped an alert it also called false_positive would be noticed -- nothing in code checks that consistency, live or offline.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Two alerts belong in the same case only when the evidence connects them causally -- a shared entity, a close timestamp or one matching indicator value is not a link on its own. And a calm or routine-sounding report is not evidence of a false positive.

src/prompt.py -- SYSTEM, stated as ordinary SOC judgement the model must apply, not as a hint about this corpus. Prompt-level ONLY: nothing in src/triage.py or src/app.py re-checks a grouping, a disposition or a citation before the UI renders them.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• IT IS NOT A CODE-ENFORCED CHECK. Both rules live only in src/prompt.py's SYSTEM text -- nothing in src/triage.py or src/app.py recomputes them. See fails.
• It does not verify that a citation is relevant, only (offline, in the eval) that it exists.
• It does not make the disposition or the grouping correct -- see Eval.taxonomy for what this run measured, not what any guardrail guarantees.
• It is not a defence against hostile or fabricated alert text -- no red-team run exists for this kit (see the security page).
• It is not a correlation ENGINE. This kit judges the bundle it is handed; which alerts get bundled together is decided upstream and is not guarded here at all.

WatchedWhat is watched, and why that one

The kit measures 18 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

15 measured by the latest run 3 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

1 recorded run. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 15 chips that all say so.

DeviationsWhat deviated

0 breaches across 1 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-score on any change to src/prompt.py or tools/build_corpus.py -- the first changes what is asked, the second changes what is asked ABOUT. Re-run evals/run.py (paid) on any change to MAX_TOKENS, and read finish_reason on every record before believing any rate the re-run reports.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library -- requirements.txt is a comment-only file explaining that the emptiness is load-bearing. The whole triage decision is three files: src/prompt.py, src/triage.py and src/adapters/__init__.py.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph. The pipeline has one path per window -- load, assemble, prompt, call, parse -- with no branching and no state carried between windows. A framework would add an orchestrator to a for-loop.

The other sideWhat a framework costs you

• A third disposition value this kit's fixed two-value vocabulary does not name (a 'needs enrichment' state, say) needs a hand-written change in src/prompt.py plus new corpus logic in tools/build_corpus.py, rather than being configured declaratively.
• No built-in retry/backoff beyond src/adapters.py's own bounded retry -- a framework's queue and worker model is not here, so a real deployment adds its own scheduling.
• No built-in observability beyond what evals/run.py prints and writes to results/ -- a framework's tracing/dashboard integration is not here. What IS recorded, and mattered, is finish_reason and reasoning_chars per call.
• No token-budget management. A framework that sized max_tokens against observed output would have caught the truncation this kit had to find twice by hand; the stdlib-only design has one constant and a comment.
• No built-in output validation beyond src/prompt.py::parse()'s tolerant-but-not-creative JSON parse -- a framework with a schema-and-consistency layer might catch a model that groups an alert it also called false_positive before it renders; this kit does not check that in code (see Guardrails.fails).

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-it-sectriage on the fast tier, 2026-08-20. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. This is the only run on record, so there is nothing to move against. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — windows are cut from the log stream per run.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT (this repository's own licence) Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineDisposition exact match, pair-wise false correlation, and citation-existence checks

Does each alert's true_positive/false_positive call match gold? Does the model avoid calling a gold true positive a false positive (missed_true_positive), overall and on the 3 deliberately mundane-worded phishing reports? Does it avoid putting two alerts in one case that gold keeps apart (false_correlation), overall and on the 8 planted coincidental-indicator pairs? Does every citation name an indicator that actually exists on an alert in that case?

MethodHow the test was run

evals/scoring.py::score, in-process, no key and no model. The same function evals/baseline.py and evals/run.py both call -- a baseline and a real run scored by two different scorers cannot be compared honestly.

The inputOne real row, seen by every grader

The formulaWhat it computes

disposition_accuracy = correct dispositions / answered, over answered=82. missed_true_positive = a gold true_positive the model called false_positive, over the 52 gold true positives; its trap subset uses exactly the 3 alert ids gold's trap_alert_ids names. false_correlation is counted over PAIRS: a gold-different pair is any two alerts not in the same true-positive case together in gold (including a true positive paired with a false positive), giving 47 pairs; its trap subset is exactly the one planted trap_pair per false_correlation window, 8 of them. citation_validity = citations whose (name, value) exists on an alert in that case / all 109 citations drafted.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's derive_gold(), which turns each alert's construction-time disposition and case_id into the window's alert_dispositions and case_groups -- re-derived independently via --verify against the windows and gold actually written to disk, asserted zero drift.

How tight can the band be? There is no tolerance band -- every field is an exact match (a two-value disposition, a grouping compared pair by pair, an indicator string looked up exactly after normalising 'k=v'/'k: v' spacing and case). Nothing here is a continuous quantity to round.

Cadence: Re-score on any change to src/prompt.py or tools/build_corpus.py -- the first changes what is asked, the second changes what is asked ABOUT. Re-run evals/run.py (paid) on any change to MAX_TOKENS, which this kit has already had to do twice.

The decisionWhen to reach for it