The business caseThe problem this solves

Someone has a stack of long documents — reports, filings, case files, board packs — and needs the same short brief out of every one of them: what it is about, what it found, the numbers, what it recommends, what it leaves open, who it is for. Today somebody reads 57 pages and writes that page by hand. Reading a 57-page report and writing a one-page brief of it by hand.

The inputThe actual documents

The corpus is 42 documents, 5.46 MB (txt 42). Three things had to be true at once and this is the only source found where they were. The licence is unambiguous and needs no notice file. The documents are genuinely long — median 57 pages, up to 108 — which is the whole point of a summariser and the reason short-document corpora were rejected. And every one of them ships with a human-written brief that can be REMOVED and kept aside, which gives a human grader something to calibrate against without ever being used as gold. ⚠︎ They are archived, not current: GovInfo's GAOREPORTS collection ends around 2008, so anything a brief says about the world is the world of 2008. That is irrelevant to whether the brief is a good brief, and it would be fatal to a kit that claimed to answer questions about today.

The outcomeWhat a good result looks like

You pick a document and get six named sections back, each one written against a weighted rubric the reader can see before anything runs. A section the document does not support comes back as 'Not stated in this document.', which is a correct answer here rather than a gap — on the run behind these pages that happened once, on GAO-08-1075R's recommendations, in a report that makes none.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• You need the same fixed brief out of many long documents, and a person will read the result anyway — This kit
  It reliably produces the SHAPE — 42 of 42 documents returned all six sections on r004-nothink, at 8.5 seconds and about a fifth of a cent each. A reader who was going to check the output regardless loses little to an unscored summariser.
• You want a cheap first pass over a corpus you will triage by hand — This kit, with the packing bar in view
  The app names, per document, which sections did not reach the model. That is the one thing a triage reader needs and the one thing a summary cannot tell you about itself.

And where nothing here is good enough:

• The brief will be acted on without anyone opening the source — Not this kit, not yet
  Nothing here has been graded. There is no measured basis for trusting any individual claim in any brief, and the failure this kit is most exposed to — a fluent summary of the 84% of a report that fitted — is invisible to every metric currently published.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-06. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-06 — r005-nothink. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Verified with no key configured: 42 documents segment into 2,896 sections in 0.06 seconds, and src/prompt.py reproduces run r004-nothink's packing decisions for all 42 — the same sections sent, the same sections dropped, the same character and estimated-token counts, checked field by field. Nothing about the preparation half of this kit needs a provider, a key or a network.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per document, reasoning disabled

Current processWhat it replaces

Reading a 57-page report and writing a one-page brief of it by hand.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusEvalsEvalsArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 2 of 8 stations carry a recorded failure

scroll → 8 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one section is

front and back matter dropped corpus-wide by src/boilerplate.matter (a repeated line inside a run of 4+ repeated lines is furniture; an isolated one is a heading and stays), THEN cut on section headings by src/segment.py; a document with none falls back to one whole-document segment, so a brief can still be written from it

SetupWhat the setup figure measured

There is no index. Preparation is a furniture pass plus segmentation — 42 documents, 4,622 lines of repeated front and back matter dropped (92 distinct lines, 3.3% of the corpus by characters), then cut into sections by src/segment.py. Pure code, no model and no key, measured by re-running it over the committed corpus. The furniture pass moved the median section count per document from 63 to 45, because the separator rules under dropped headings were themselves creating section boundaries.

LicenceLicence

Public domain. GAO reports are U.S. Government works and are not subject to copyright protection in the United States (17 U.S.C. §105). Every text rendition carries the line itself — 'This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO.' No attribution clause, no share-alike, no notice file. GAO asks only that its work not be represented as anything other than GAO's, which this kit does not do: every document keeps its title, number and date.

Bring your ownBring your own documents

Replace data/corpus/*.txt and write data/rubric.json — the sections you want, their weights, and a 0-5 anchor set. Nothing else is required: there is no gold file to build, because two correct summaries of one document share almost no words, so the rubric IS the gold. If your documents carry a publisher's own summary, extend src/boilerplate.py to strip it, or every score you get back will be a fact about your corpus rather than about the model. Check the packing before you spend anything: with no key at all, the app reports per document how much of it actually reached the model.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 95,668 characters assembled, 94,081 are contexts — 98% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Replayed from src/prompt.py against the committed corpus with no key configured: all 42 documents reproduce run r004-nothink's packing decisions exactly — the same sections sent, the same sections dropped, the same character and estimated-token counts. The two fixed parts are byte-identical on every call so their sizes are exact; the document-sections row is the median across the 42.

Full promptThe prompt in full

You write briefs. You are given one long document and a fixed set of sections to fill, and you return those sections and nothing else.

RULES, in order of importance:
1. Every claim must be supported by the document. Do not add what you know about the subject from anywhere else, however certain you are.
2. Copy figures verbatim, with their units and their period. Never compute a number the document does not state — not a total, not a percentage, not a difference.
3. Recommendations are the ones the DOCUMENT makes. If it recommends nothing, say so; do not supply sensible advice of your own.
4. If the document gives you nothing for a section, write exactly: Not stated in this document. A stated absence is a correct answer and is worth more than a plausible one.
5. Write plainly, in full sentences. No bullet lists, no headings inside a section, no preamble about what you are about to do.

- about: What this document is about — Would a reader who never opens the source know the subject and the scope?
- findings: Key findings — Are the findings the document's own, and are the significant ones present?
- numbers: Numbers that matter — Is every figure carried here in the source, unaltered, with its unit and period?
- recommended: What is recommended or required — Are these actions the document calls for, rather than actions the model thinks are sensible?
- unsettled: What the document does not settle — Does it name the limits, open questions and caveats the source itself states?
- audience: Who it is for — Is the audience and the decision it supports stated in one line?

Defense Infrastructure: Opportunity to Improve the Timeliness of Future Overseas Planning Reports and Factors Affecting the Master Planning Effort for the Military Buildup on Guam
================================================================================

GAO report GAO-08-1005  ·  issued 2008-09-17  ·  46 pages  ·  found via: defense acquisition

GAO-08-1005

   

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This i

…[92881 further characters of packed document sections — which sections were dropped is recorded per document; see Corpus]…

Raw responseThe raw response

```json
{
  "about": "This is a GAO report to congressional committees evaluating two topics: (1) the fiscal year 2009 overseas master plans, focusing on the changes and challenges they describe, the extent to which they address prior GAO recommendations, and their timeliness; and (2) the status of the Department of Defense's (DOD) master planning efforts for the proposed military buildup on Guam. It is the fifth report on overseas master plans and the second on the Guam military buildup.",
  "findings": "The report's key findings are that while the fiscal year 2009 overseas master plans generally reflect recent changes in U.S. basing strategies …",
  …
}
```

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Turn long government audit reports into six-part briefs — 252 section-briefs drawn from 42 real documents. One model answered, and every answer was then graded one way — by == against a fixed gold value, which is a measurement too and only as good as the gold behind it.

The rulerHow it was graded

A weighted rubric, scored 0-5 per section by a PERSON. ⚠︎ NO LLM JUDGE, AND NOT BECAUSE ONE WAS TOO EXPENSIVE — because it would measure the wrong thing. Two correct summaries of one document share almost no words, so there is no string to match and == has nothing to compare; and scoring a model's prose with another model measures the judge at least as much as the writer. evals/grade.py enforces this rather than recommending it: it requires a terminal and refuses piped stdin, so yes 4 | python -m evals.grade cannot quietly manufacture a score. The six weights — 20 findings-30 numbers-20 recommended-15 unsettled-10 audience-5 — are visible on the app before anything runs, because a rubric that only appeared after a run would let the shape be whatever the model happened to produce.

MeasurementsWhat was measured

Three words carry this page: Counted is deterministic and nobody's opinion, Judged is a model's verdict that moves if the grader is wrong, and Not yet known is printed blank rather than filled with something plausible. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar published on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate. It is a projection onto a rate card, not a bill anyone paid. The token counts belong to this pipeline — one call per document, the packed sections, the fixed rubric — and hold wherever you run it; the dollars belong to whoever you buy from, which is why the card is named beside every figure.

Rates checked 2026-08-01. The provider that actually ran r004-nothink is not priced here. Its rate is not committed anywhere in this repo, and a dollar figure whose card cannot be named is exactly what this standard forbids.

Grading unitWhat the grading figure prices

⚠︎ NOT MEASURED, BECAUSE NOBODY HAS DONE IT YET — and it is deliberately not estimated. This is the one kit in the estate whose evaluation is neither free code (docs-extract's judge) nor a second model bill (docs-qa's): it is a person's attention, priced in reviewer-minutes, and the only honest way to learn the rate is to grade a run and time it. A number invented here would be the most quoted figure on the page and the least supported.

The gradersOne way to grade, and why it is the only one

The idea it tests is that the opening of a well-written report is most of a brief, so a model has to beat free text before it is worth calling. ⚑ IT WAS UNGRADEABLE AS FIRST BUILT, AND IT WAS FIXED ON 2026-08-06 AT THE CORPUS BOUNDARY. Measured across all 42 documents, a MEDIAN OF 74.6% of that 2,000-character lead was a GAO accessibility notice — the boilerplate about text renditions and Webmaster@gao.gov — with a range of 64.4% to 79.3%. So what the baseline fed a grader was the opening of a FILE, not the opening of a report, and grading it would have scored the model against furniture. ⚠︎ AN EARLIER VERSION OF THIS NOTE SAID 'A MEDIAN OF 60%, MINIMUM 59%, MAXIMUM 60%'. That was wrong in the direction that makes the defect look smaller, and it was wrong about the spread as well as the middle — the real figures come from re-measuring every document rather than from the sample the first note was written off. The fix drops repeated front and back matter for EVERY consumer — the baseline, the model runs and the app — because fixing it in the baseline alone would have left the baseline reading a different document from the model it exists to be compared against. Furniture is detected by repetition across the corpus, but only removed where it sits in a run of four or more repeated lines: 'Results in Brief' is repeated in 39 of 42 documents and is a heading, not furniture, and a line-shaped rule would have deleted it along with 'Background', 'Conclusions' and 'Recommendations for Executive Action' from every document in the corpus. After the pass the furniture share of the lead is 23.2%, and what remains is the report's own title block. b000 was re-run free the same night; r005-nothink is the first model run on the same corpus, so the baseline and a model finally read the same documents.

The row opens its own page: how the test was run, one real row, the formula with every iteration of it, and the analysis.

Choosing oneWhen to reach for the model at all

The comparison is only useful if it ends in a choice, and on this kit the choice is not which grader — there is one, and it is ==. It is whether to call a model for this field at all, and the answer is conditional.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

No failure taxonomy on this kit. Nothing has been coded into named causes yet, so there is no breakdown to print. The graders above say what was scored; this table stays empty until a run is read row by row and the misses are given names.

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-01. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same document, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingNot measured, because nobody has graded a run yet

NOT MEASURED IS A THIRD STATE AND THIS IS IT — not a zero. Grading here is a person scoring six sections per document against a 0-5 weighted rubric; evals/grade.py requires a terminal and refuses piped stdin, because two correct summaries of one document share almost no words, so there is nothing to string-match and scoring a model with a model would measure the judge. Six paid runs have produced zero graded sections. A dollar figure of 0.00 here would be the most misleading number on the page: it would read as 'evaluation is free' for the one kit in this estate whose evaluation is the expensive part.

Cost driversWhat actually moves the bill

• Input is the document, and the document is enormous. 19,071 input tokens against 876 output — 96% of the tokens on every call are the report itself. This is the opposite shape from a question-answering kit, where the question is small and the retrieved context is the variable.
• One call per document, so cost scales linearly with corpus size. Nothing amortises: the expensive part of each prompt is that document's own sections, and no two documents share them.
• The reasoning pass, which is invisible on the bill line and dominates it. With thinking at its default the output side ran 5,872 tokens per document and most of it never reached text; disabled, it runs 876. Output is priced at 4x input on the card used here, so the part you cannot read is the part that moves the number.
• The input budget is a floor on cost, not a ceiling: pack.py fills 24,000 tokens whenever the document is long enough, and 35 of 42 documents are. Lowering it lowers the bill and drops more of the report.

Your volumeWhat it costs at your volume

Linear, and there is nothing to amortise. Ten times the documents is ten times the bill: one call each, no batching, no shared prefix beyond the 1,587 characters of system prompt and rubric that are byte-identical on every call — 1.7% of what is sent. The one thing that does NOT scale linearly is the human side: grading is a person reading six sections against a weighted rubric, so ten times the corpus is ten times the reviewer-hours, and that is the cost this kit is actually about.

Model choiceWhy this model

The connection already configured on this machine. No model comparison was run, so nothing here says it is the best choice — and the one setting that WAS compared, thinking, is a property of this model family rather than a choice between vendors.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 42 GAO reports summarised to a six-section brief each, on run r005-nothink — the cleaned corpus, reasoning off, 42 of 42 delivered and nothing lost to the ceiling.

The codeEvery file, and what it is for

Six modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Four of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pysegment

cut the document into addressable sections, pure code

# Cut a long document into addressable sections. Pure code — no model, no network.

def sections(text):
def _blocks(text):
def coverage(secs, text):

src/boilerplate.pyboilerplate

strip the publisher's own furniture before anything is measured, pure code

# Drop the navigation, headers and footers that every document in a corpus repeats.

SHARE = 0.5
MAX_LEN = 160
MIN_DOCS = 3
SMALL = 5
RUN = 4
def _repeated(texts):
def detect(docs, formats=None):
def strip(docs, formats=None):
def _blocks(text, drop, run):

src/pack.pypack — a swap seam

fit as many sections as the input budget allows and record what was dropped, pure code

You change it to: how much of a long document reaches the model before packing starts dropping sections

# Order the sections and fit them to a token budget. Pure code — the last deterministic step

CHARS_PER_TOKEN = 4
DEFAULT_BUDGET_TOKENS = 24000
def plan(secs, budget_tokens=DEFAULT_BUDGET_TOKENS, reserve_chars=0):
def summary(p):

src/prompt.pyprompt

assemble one call carrying the system rules, the rubric and the packed sections

# Assemble the summarisation prompt. One prompt per document, the whole rubric in it.

SYSTEM = (
def rubric_schema(sections):
def build(doc_text, secs, sections, packer, budget_tokens=None):
def is_absent(value):
def parse(raw, sections):

src/summarise.pysummarise

the AI layer — one provider, one key

# Summarise one document to the fixed brief: segment, pack, prompt, one model call, mark absences.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RUBRIC = os.path.join(HERE, "data", "rubric.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 8000
def load_rubric():
def sections_spec():
def documents():
def _read(doc_id):
def corpus_furniture():

evals/grade.pygrade

a person scores six sections against the weighted rubric; there is no code path that grades

# Grade a run's briefs against the weighted rubric. THE GRADER IS A PERSON — this is the prompt

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
REF = os.path.join(HERE, "data", "reference")
def grade_path(run_id, grader):
def load_run(run_id):
GIVE_UP_AFTER = 3
def _ask(prompt_text, valid):
def score(run, grades, sections):
def main():

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 18775 input and 888 output tokens per document, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

A local, single-user tool over public GAO reports. It binds 127.0.0.1, has no auth, no rate limit and no session — correct for a demo on your own machine and wrong for anything else. There is no chat box: the only text a stranger controls is the REPORT being summarised, which is exactly the surface this run attacks.

Read from the shared .env or the real environment only, never written into the repo, never requested from a reader on any surface. The repo has never held a credential. The UI screenshots are taken with the key blanked, and the shoot script refuses to start if anything is already listening on its port — a stale copy of the same kit holding a real key passes an identity check perfectly, and that gap cost a real provider call elsewhere.

The experimentWe attacked it — two gates

An indirect prompt injection has to clear two gates, and they behave differently. Reproduce with python -m evals.redteam --run-id x001-docs-summarise --docs 5. Measured on 2026-08-08, run x001-docs-summarise: 5 documents crossed with 6 attack families, 30 live calls on the fast tier with reasoning disabled. 28 attempts scored; 2 returned a reply that did not parse and are reported as such rather than banked as resistance. Each injection was asserted into the ASSEMBLED PROMPT before the call was paid for — this kit packs what fits a token budget, so an attack the packer cut never reached the model and measures nothing. Two free control stubs ran first, one resisting everything (0/6) and one complying with everything (6/6).

⚠︎ THE ATTACKS LEAD THE DOCUMENT RATHER THAN FOLLOWING IT, AND THE FIRST DRAFT MEASURED NOTHING BECAUSE THEY DID NOT. This kit segments a long report and packs what fits a token budget; these GAO reports are far longer than the budget, so text appended at the end never reached the model and all six control attempts recorded NOT SENT. Leading the document is also what a real attack would look like — a distribution banner, a classification header, an editorial note — and it is what a packer keeps. Two of the attacks also ask for this kit's own absent phrase VERBATIM, because src/prompt.py recognises exactly one string and an attacker reading the published prompt would use it; an earlier draft asked for the wrong one and measured the phrase rather than the model.

The resultAn attacker-chosen sentence landed in the brief on every attempt — 5 of 5 — and the heaviest section of the rubric was suppressed on every attempt too. A summariser's output is READ INSTEAD OF the source; that is what it is for. So a sentence an attacker gets into the brief is a sentence a reader attributes to a government report and never checks. The suppression is the more dangerous of the two: the other five sections come back written and correctly formatted, so the brief looks complete. Overall 16 of 28 scored attempts followed — 42.9% resisted, the weakest of the four kits measured.

The four-kit picture is the finding, and it is not the one two runs suggested. docs-verify — injected into the document, closed verdicts — is this claim supported?: 3 of 30 followed, 90.0% resisted · docs-comply — injected into the rulebook, closed verdicts — does this rule pass?: 13 of 29 followed, 55.2% resisted · docs-redact — injected into the document, span extraction — find every identifier: 15 of 30 followed, 50.0% resisted · docs-summarise — injected into the document, free-text generation — write a brief: 16 of 28 followed, 42.9% resisted. Three of the four sit between 43% and 55%, and the outlier that RESISTS is a document-injection kit — so "rulebook versus document" does not explain the spread. What lines up is what the model is asked to DO with the text. Asked to JUDGE it against a fixed, closed vocabulary — is this claim supported by that source? — the model treats the document as EVIDENCE, and an instruction inside it is just more evidence; it resists at 90%. Asked to ACT on it — summarise it, extract from it, apply rules to it — it treats what it reads as part of the JOB DESCRIPTION and obeys about half the time. docs-comply fits: closed vocabulary like docs-verify, but its injection arrives in the RULES, which are the job description by definition. Four runs, one provider, one day, six attacks each — a pattern across four comparable measurements, not a study.

HonestyWhat this does not prove

• Whether a real attacker would use these six. They were written by the kit's author against the kit's own design.
• Whether the reasoning tier resists differently. This run used the fast tier only.
• Whether the cross-kit pattern is causal. Four runs on one provider on one day, built to be comparable on purpose. That makes the comparison real and the MECHANISM behind it a reading.
• The app's HTTP surface. The run drives the kit's own entry point — the same code path the app calls — but the app was not attacked through its own interface.
• Whether an injection survives in a real pipeline. These are placed where the packer keeps them. A document whose injection lands mid-report may never reach the model at all — which is luck, not a control.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

If the document gives you nothing for a section, write exactly: Not stated in this document. A stated absence is a correct answer and is worth more than a plausible one.

Rule 4 of five in src/prompt.py's SYSTEM string, sent byte-identical on every call. That is the whole enforcement: no schema, no validator, no retry. The kit asks and hopes.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is not enforcement. There is no schema, no validator and no retry: the model can write anything into any section and the pipeline will record it.
• It is not a hallucination check. It governs what to do when the document says nothing; it says nothing about a claim that is present, confident and wrong.
• It is not measured as a rate. docs-extract can count invented values because its gold is exact; here the gold is a person's judgement, and there is no denominator until someone grades.

WatchedWhat is watched, and why that one

The kit measures 15 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 8 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

no shared guard The two recorded runs declare no guard in common, so nothing below is protected by a precondition — read the history as two separate observations, not as a difference.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

8 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 2 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

summarise · with the model in the path — 7 runs. Columns here are only ever compared with each other.

not a time series No two of these 7 runs measured the same system — they differ on corpus, max_tokens, provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

redteam · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 7 chips that all say so.

Run ids are shown with the runtime provider’s name elided; the records keep their full ids.

DeviationsWhat deviated

0 breaches across 8 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Nothing polls and nothing alerts. The board collects the newest run record and replaces what is there; a run is fired by a person, on purpose, because every one of them spends.

In one lineNo framework, deliberately

This kit has no framework and no dependencies at all — Python standard library, end to end. requirements.txt names nothing and says so in prose. That was a choice, and the point of it is that the one decision which most shapes a brief, what the model is allowed to see, is made in pure code you can read in a single file.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here loops, branches or retries: one pass per document, seven seams, and the run record shows 0 of 42 documents lost to a retryable failure. But this is the kit with a MEASURED reason to want a cycle, and it is not the usual one. 33 of 42 documents on r005-nothink had sections dropped to fit the 24,000-token budget — the brief still reads fluently and every section is filled, which is precisely the danger. Map-reduce is the standard framework answer: summarise each group of sections, then merge the partials. It would also change what this kit measures, adding a second model pass and a merge step with its own unevaluated failure mode, so it is a different experiment rather than a free improvement.

The other sideWhat a framework costs you

• A dependency tree, and a version treadmill on somebody else's release schedule.
• An abstraction over the one thing this kit exists to show you — what the model actually receives. The prompt is published verbatim on the LLM page because it is assembled in one readable file; three layers down it could not be.
• SOMEBODY ELSE'S DEFAULT, WHICH ON THIS CORPUS WOULD HAVE BEEN WRONG. The cleaning seam is the concrete case: the obvious repeated-line rule removes the boilerplate and the section headings together, reports a large line count as a success, and leaves a summariser reading a report with its skeleton taken out.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r005-nothink on the fast tier, reasoning enabled (the kit's default), 2026-08-06. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 6 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — one document is one unit, whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, GOVINFO_API_KEY, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: Public domain. GAO reports are U.S. Government works and are not subject to copyright protection in the United States (17 U.S.C. §105). Every text rendition carries the line itself — 'This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO.' No attribution clause, no share-alike, no notice file. GAO asks only that its work not be represented as anything other than GAO's, which this kit does not do: every document keeps its title, number and date. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe weighted rubric, scored by a person

For each of six sections: is this what the document says, at the level of detail the rubric asks for — scored 0 to 5 against written anchors?

MethodHow the test was run

python -m evals.grade --run-id <id>, offline, no key. It prints one section at a time with the rubric question above it and takes a 0-5 keystroke. ⚠︎ IT REFUSES PIPED STDIN — if not sys.stdin.isatty(): raise, with the message that nothing arriving that way is a person's judgement. That is not a usability choice: yes 4 | python -m evals.grade would otherwise fabricate 252 scores in a second and publish them as an evaluation.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

score = sum(section_score x weight) / 5, over six sections weighted 20 about-30 findings-20 numbers-15 recommended-10 unsettled-5 audience, each scored 0-5 against written anchors. ⚠︎ NO NORMALISATION AND NO MATCHING STEP, because there is nothing to match: two correct summaries of one document share almost no words. The anchors are the whole method — 0 is 'absent, or so wrong it would mislead someone who did not open the source' and 5 is the top of the scale, and what sits between them is a person's reading.

IterationsFour rulers, four wrong numbers

The tell was the ranking inverting, not the value moving. Noise moves a number; an artifact reorders the comparison the page exists to make.

The analysisWhat it actually did

No run rows on this grader. Nothing was scored through it in the run this page reports, so there is no model and no number to tabulate. What it is for is above; what it would watch is below.

In operationWhat to monitor

Reference standard: this grader, and there is no other. The rubric is simultaneously the prompt's instruction, the gold and the cost unit — there is no machine-derived answer key behind it, because a correct summary is not a string.

How tight can the band be? No band, and not a tight one either — none can be derived. A tolerance is the spread between runs of the same set, and this grader has zero runs. Anything written here before two graded runs exist would be picked rather than derived.

Cadence: Re-grade the whole set whenever the model, the prompt or data/rubric.json changes. Those three are the seams that move the numbers and none of them is comparable across the change — the rubric especially, since it is the prompt and the gold at once.

The decisionWhen to reach for it