A single log line carries almost no information, so the unit of work is a window of them and choosing that window is the whole design.
ConceptWhat it is
Logs and events are the highest-volume source most organisations have and the least like prose. A line is terse, templated, and only meaningful next to the lines around it. Nobody reads one; people read a burst.
That changes the unit of work. For documents the natural unit is a passage; for logs it is a window — a time slice, a request trace, an incident. Choosing that window is the substantive decision, and it is made before any model is involved.
How it worksThe mechanics
Records are parsed into structured fields and grouped into the chosen window, then summarised or templated so that repetition collapses. A thousand identical timeout lines carry the same information as one line and a count, and sending the thousand wastes both context and money.
Because volume is the defining constraint, sampling and retention are design decisions rather than operational afterthoughts. What gets indexed is usually a small, deliberately biased selection — errors, anomalies, sampled successes for comparison — and the bias has to be recorded, because it shapes everything measured afterwards.
At a glanceSee it
The window is the unit, not the line. Collapsing repetition and recording the sampling bias are what keep the corpus both affordable and honest.
When to use itWhere it fits
- Incident triage and diagnosis, where the question is what happened across a burst.
- Anomaly explanation, where a model reads a window a human would otherwise have to scan.
- Any system whose unit of work is genuinely a window rather than a document.
- As a feed into the improvement loop, since production traces are themselves an event stream.
When NOT to use itLimits & anti-patterns
- As raw lines fed to a model, which is expensive, noisy and usually worse than a query.
- For questions a structured query answers exactly — counting errors does not need a language model.
- Where personal data flows through logs unredacted, which is common and rarely audited.
- When retention is short enough that the corpus disappears before anyone asks the question.
Trade-offsAdvantages & costs
Advantages
- Enormous coverage of what the system actually did, rather than what anyone believed it did.
- Already collected in most organisations, so ingestion is a routing problem rather than a new capture.
- Naturally timestamped and ordered, which makes windowing straightforward.
- Templating collapses volume dramatically, often by more than an order of magnitude.
Trade-offs & costs
- Volume makes cost the first-order concern in a way it is not for documents.
- A single record is close to meaningless, so the window choice determines quality entirely.
- Personal and secret data leaks into logs routinely and needs redaction before indexing.
- Sampling bias propagates silently into every measurement made downstream.
ExampleIn the real world
An operations assistant is given a window of logs around each alert. The first version passes the raw window and one noisy service dominates every context with repeated connection warnings, crowding out the three lines that explain the failure. Templating the repeats into a single line with a count left room for the signal, and cut the cost per question by most of it.
ToolsHow to implement it
- Drain or a log-template minercollapsing near-identical lines into a template plus a count.
- OpenTelemetrytrace ids that define a window far better than a time slice does.
- Vector or Fluent Bitrouting and redacting at collection time, before anything is stored.
- A recorded sampling policyso the bias in the corpus is a known quantity rather than a surprise.
Cost & effortWhat it takes
The one source where corpus cost can exceed inference cost. Storage and indexing dominate, and the effective lever is collapsing repetition rather than buying capacity. Engineering effort is moderate; the window definition is the part that deserves the thinking, and it is usually revised at least once after seeing real failures.