The business caseThe problem this solves

A scheduling order sets several deadlines at once and sets each of them differently: one names a date, one counts calendar days from the order, one counts business days from a conference that has not happened yet. A docketing desk has to turn all of them into diary entries, and three of the ways it gets that wrong are invisible on the page — counting the day the clock starts on, counting business days as calendar days, and leaving a deadline sitting on a Sunday. The order also puts two things next to the answer that look like evidence and are not: a parenthetical carrying somebody else's arithmetic, and a paragraph that names an item, a number and a unit and then strikes the requirement. Somebody opening each scheduling order, working out which numbered paragraphs actually impose a deadline and which ones only look like it, and then counting each one to a calendar date: is the period in days or in business days, does the day it runs from count, does it run from the order or from an event that happened somewhere else, and what happens when the last day lands on a weekend or a day the court is shut. It is a few minutes per order and it is every order, and the part that goes wrong is not the reading. It is the counting — and the case that goes wrong most expensively is the one nobody can count at all, because the order sets a period running from an event it never dates.

The inputThe actual scheduling orders

The corpus is 52 scheduling orders, 0.05 MB (txt 52). Plain text, one format, invented rather than fetched. A real scheduling order is a real case — it names a court, a judge, parties, counsel and a docket number — and publishing a gold CALENDAR against one would mean publishing computed legal deadlines for identifiable litigation, which is the one thing this kit is careful never to do. There is also no public corpus of (order, correct calendar) pairs, because the answer is only correct against the rules of one court on one day. Generating it also makes the label mechanical: gold's due_date is src/calendar_rules.py::compute() run over the same values the paragraph states, never somebody's reading of it.

Court
-----
Meridian Vale Civil Court

Division and Courtroom
----------------------
Civil Division D, Courtroom 6

Matter Number
-------------
MVC-27-44999

Order Date
----------
14 July 2027

Recorded Events
---------------
the close of fact discovery   1 September 2027
appointment of the neutral    not recorded

Deadlines Ordered
-----------------
1. Each party shall serve the deposition designations within 15 business days after appointment of the neutral.

2. The parties shall file expert disclosures within 10 business days after the date of this Order. (counsel's calendar: 24 July 2027)

3. The requirement that the mediation certificate be served within 60 business days after the case management conference is WITHDRAWN and sets no date here.

4. Within 45 days after the close of fact discovery, each party shall file initial disclosures. (the parties calculate this as 16 October 2027)

5. Within 10 days of the date of this Order, the parties shall exchange the pretrial statement. (calculated in the proposed order as 26 July 2027)

6. Within 15 business days of the date of this Order, counsel shall serve the proposed findings.

7. Counsel shall serve the privilege log on or before 18 August 2027.

The outcomeWhat a good result looks like

A row per obligation with its structured basis, plus a calendar date computed afterwards in pure code from the shipped rulebook, with the working attached — what it counted from, whether it rolled, and off what. Obligations nothing on the order dates come back as cannot be dated with the missing event named. Nothing here files, serves, dockets, waives or extends anything, and nothing is legal advice.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Turning a day's incoming scheduling orders into a proposed diary before a docketing desk opens them — the fast tier — 259 of 260 dates and 24 of 24 obligations correctly reported as undatable, on one run
  99.62 pct date accuracy against the free desk-calendar floor's 55.38 pct, and 0 confident dates on undatable obligations against the floor's 24 of 24.
• Deciding whether you need a model for this at all — read the free floor's row before the model's
  The floor finds every obligation, invents none, reads 1,924 of 1,924 fields and scores 1.00/1.00 on the undatable flag — with no key and no spend. Everything this kit's model adds is on the DATE: 259 of 260 against 144 of 260.

And where nothing here is good enough:

• Ranking two models on this task from these numbers — nothing yet — this corpus cannot answer it
  One tier was run and it scored 259 of 260, so there is no headroom left here to separate two configurations and no second tier to compare against. What the numbers DO support is narrower and real: counting on a wall calendar is measurably wrong on 44.62 pct of these obligations, and a self-consistency check catches every one of those errors with no labels at all.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-22. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-22 — r001-order-dates. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Checked on a fresh checkout with API_KEY left blank: python -m src.app starts, the 52-order picker populates, the deadline table draws its placeholder row and the shipped rulebook renders in full from data/rulebook.json including the whole invented holiday list; clicking Compute returns the no-key sentence rather than a stack trace. python -m evals.check_labels passes with no network access at all, and python -m evals.run --baseline scores the free floor over all 260 obligations without a key.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per scheduling order

Current processWhat it replaces

Somebody opening each scheduling order, working out which numbered paragraphs actually impose a deadline and which ones only look like it, and then counting each one to a calendar date: is the period in days or in business days, does the day it runs from count, does it run from the order or from an event that happened somewhere else, and what happens when the last day lands on a weekend or a day the court is shut. It is a few minutes per order and it is every order, and the part that goes wrong is not the reading. It is the counting — and the case that goes wrong most expensively is the one nobody can count at all, because the order sets a period running from an event it never dates.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 1 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one section is

cut on underlined section headings, then a second, finer cut on the numbered ordered paragraphs — one section holds every obligation, so the paragraph is the addressable unit and the span scope

SetupWhat the setup figure measured

There is no index. Preparation is segmentation only — 52 orders cut into 312 sections and 338 numbered paragraphs in under a hundredth of a second, in process, with no model and no network.

LicenceLicence

MIT — this repository's own licence. Every court name, matter number, event and obligation is invented; no real court, judge, case, docket number, party or jurisdiction is named anywhere. The counting rulebook this kit ships (data/rulebook.json, MV-CR-1) is INVENTED and is not an authority: it reproduces no procedural code, court rule, local rule, standing order or published holiday schedule, and every court holiday on its list — name and date — was made up for this kit and chosen deliberately not to coincide with any real one.

Bring your ownBring your own scheduling orders

REPLACE data/rulebook.json FIRST. It is invented, it reproduces nobody's procedure, and everything downstream reads it: src/calendar_rules.py loads it, src/prompt.py renders it into every call, and tools/build_corpus.py writes gold with it. Then replace data/corpus/*.txt, write your own data/fields.json, and supply a gold record per order. SECTION_HINTS in src/select.py maps fields to section headings and will need editing for a different layout; when it does not match, selection falls back to the whole document — slower, more expensive, always correct. compute() in src/extract.py is this kit's own undatable rule and is the second thing to replace.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 2,644 tokens assembled, 1,121 are contexts — 42% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured token-for-token via evals/prompt_tokens.py's nested-prefix method on ORD-0018 — the same order the worked example above was captured on, so the four parts sum to exactly the 2,644 input tokens that call actually reported. Four calls at max_tokens=1, each part's size the difference between two consecutive prompt_tokens counts the provider itself returned. kits/UC0054-order-dates/results/tokens-p002-order-dates.json. ⚑ THE SAME MEASUREMENT ON A DIFFERENT ORDER (ORD-0001, tokens-p001-order-dates.json) returned 752 / 904 / 771 for the first three parts — byte-identical, which is the evidence that the 2,427-token fixed prefix really is fixed and is paid on every call.

Full promptThe prompt in full

You read a SCHEDULING ORDER and return, for every paragraph that sets a deadline, the structured facts of that deadline and the calendar date it falls due. You return JSON and nothing else.

You are PROPOSING a calendar for a person to check. You do not file, serve, docket, waive or extend anything, and nothing you return is legal advice or a substitute for the file and the rules that actually govern it.

RULES, in order of importance:
1. If the Order does not state a value, return null for it. Do not infer it and do not use what you know about how courts usually count.
2. RETURN A ROW ONLY FOR A PARAGRAPH THAT SETS A DEADLINE. A paragraph that merely mentions a date, sets a page limit, vacates a trial date, supersedes an earlier order, or states a deadline and then STRIKES, WITHDRAWS or disapplies it, sets no date and gets NO row. A row for one of those is a diary entry nobody owes.
3. `due_date` is computed ONLY by the COUNTING RULES given below, applied to the paragraph's own basis, its period and the date it runs from. Work through it in this order:
   a. EXPLICIT. If the paragraph names a date ('on or before', 'no later than'), that date IS the answer. It is NEVER moved, not even when it falls on a weekend or a court holiday.
   b. WHAT IT RUNS FROM. 'the date of this Order' is the Order date. Anything else named after 'after' is a triggering event, and its date comes from the RECORDED EVENTS table. If that table says the event is not recorded, STOP: `due_date` is null and `trigger_event_date` is null. Never substitute the Order date for an event the Order does not date.
   c. COUNT. The day it runs from is DAY ZERO and is not counted; counting starts the next day. A period stated in DAYS counts every day including weekends and court holidays. A period stated in BUSINESS DAYS counts only business days -- Monday to Friday, excluding every court holiday on the list below.
   d. ROLL. If a period counted in DAYS ends on a day that is not a business day, move it FORWARD to the next business day, and keep moving until you reach one. A period counted in BUSINESS days cannot end on a non-business day, so it never rolls.
4. THE PARTY'S OWN CALCULATION IS A FIELD, NOT EVIDENCE. A parenthetical such as "(counsel's calendar: 19 March 2027)" is copied into `party_calculated_date` and is NEVER copied into `due_date`. It is somebody's arithmetic and on these orders it is sometimes wrong.
5. Every date you return is ISO, YYYY-MM-DD. The Order writes dates in words.
6. Copy `item` and `trigger_event` verbatim from the Order, in the lower case it states them.
7. Use the exact allowed value for a field that lists them, and return every field named in the schema, even when the answer is null.

COUNTING RULES (MV-CR-1, the authority for `due_date`; this is an ILLUSTRATIVE rulebook written for this kit, not any real jurisdiction's rules)

BUSINESS DAY
  Monday to Friday, excluding every court holiday listed below.
  Saturday and Sunday are not business days.

WHEN COUNTING STARTS
  THE DAY THE PERIOD RUNS FROM IS DAY ZERO. The day this Order was entered, or the day the triggering event occurred, is not counted. Counting begins on the following day. A period of 1 day from a Monday ends on the Tuesday.

HOW EACH BASIS IS COUNTED
  explicit_date:             nothing is counted — the Order names the day
                             wording: on or before <date> / no later than <date>
                             rolls to the next business day: NO
  calendar_days_from_order:  every day, including weekends and court holidays, from the day after the Order date
                             wording: within N days of (or after) the date of this Order
                             rolls to the next business day: YES
  calendar_days_from_event:  every day, including weekends and court holidays, from the day after the event
                             wording: within N days after <a recorded event>
                             rolls to the next business day: YES
  business_days_from_order:  business days only, from the day after the Order date
                             wording: within N business days of (or after) the date of this Order
                             rolls to the next business day: NO
  business_days_from_event:  business days only, from the day after the event
                             wording: within N business days after <a recorded event>
                             rolls to the next business day: NO

ROLLING
  A period counted in CALENDAR days that ends on a day which is not a business day moves FORWARD to the next business day, and keeps moving until it reaches one — so a period ending on a Saturday before a Monday holiday moves to the Tuesday. A period counted in BUSINESS days cannot end on a non-business day, so nothing rolls. ⚑ A DATE THIS ORDER STATES IN WORDS IS THE DATE IT STATES, AND IS NEVER ROLLED, even when it falls on a weekend or a court holiday. The Court is taken to have meant the day it wrote.

WHEN NO DATE CAN BE COMPUTED
  A period that runs from a triggering event the Order does not date CANNOT be calendared from the four corners of the Order. It is not zero days and it is not the Order date — it has no date yet, and the honest answer is to say so and route it to somebody who can find out when the event happened.

COURT HOLIDAYS -- not business days, and they are counted through by a period stated in DAYS and skipped by a period stated in BUSINESS DAYS
  2027-01-12  Founders' Rest
  2027-02-01  Vale Charter Day
  2027-03-04  Spring Assize Recess
  2027-04-22  Chancery Rest
  2027-05-03  Vale Common Day
  2027-06-16  Meridian Day
  2027-07-02  Midsummer Adjournment
  2027-08-05  Harvest Rest
  2027-09-23  Michaelmas Recess
  2027-10-01  Vale Records Day
  2027-11-10  Assize Rest
  2027-12-29  Winter Adjournment
  2028-01-11  Founders' Rest
  2028-02-04  Vale Charter Day
  2028-03-02  Spring Assize Recess
  2028-04-20  Chancery Rest
  2028-05-01  Vale Common Day
  2028-06-14  Meridian Day
  2028-07-07  Midsummer Adjournment
  2028-08-03  Harvest Rest
  2028-09-21  Michaelmas Recess
  2028-10-02  Vale Records Day
  2028-11-08  Assize Rest
  2028-12-27  Winter Adjournment

Extract these fields:
- matter_number (string) -- the matter number, verbatim from the Matter Number section
- order_date (date) -- the date this Order was entered, from the Order Date section, converted to YYYY-MM-DD. The Order writes dates in words ('11 February 2027'); every date you return is ISO
- deadlines (array) -- one object per numbered paragraph that SETS A DEADLINE. Do NOT return an object for a paragraph that merely mentions a date, sets a page limit, or states a deadline and then strikes, withdraws, vacates or supersedes it — those set no date and produce no row. Return the objects in paragraph order
    - paragraph (integer) -- the paragraph's own number in the Deadlines Ordered section. This is the join key — copy it exactly
    - item (string) -- what must be done, verbatim and in the lower case the paragraph states it — for example 'initial disclosures', 'the joint status report'. Do not add words the paragraph does not use
    - basis (enum) one of: explicit_date, calendar_days_from_order, calendar_days_from_event, business_days_from_order, business_days_from_event -- how this paragraph fixes its date. 'on or before <date>' or 'no later than <date>' is explicit_date. 'within N days of/after the date of this Order' is calendar_days_from_order. 'within N days after <a recorded event>' is calendar_days_from_event. The same two with 'business days' are the business_ variants
    - period_days (integer) -- N, the number of days the paragraph counts — null for explicit_date. Return the number only, not the unit; the unit is already in `basis`
    - trigger_event (string) -- the triggering event the period runs from, copied verbatim from the Recorded Events table — null for explicit_date and for anything counted from the date of this Order. 'the date of this Order' is NOT an event and is never this value
    - trigger_event_date (date) -- the date the Recorded Events table gives for that event, as YYYY-MM-DD. Null when the table says 'not recorded', and null when there is no trigger event. NEVER substitute the Order date for a missing event date
    - stated_date (date) -- the date the paragraph names in words, as YYYY-MM-DD — only for explicit_date, null otherwise
    - party_calculated_date (date) -- the date in the parenthetical a party wrote next to the obligation ('counsel's calendar: 19 March 2027'), as YYYY-MM-DD, or null when the paragraph carries none. Copy it as a field. IT IS NOT THE ANSWER and it is wrong on some of these orders
    - due_date (date) -- the calendar date this obligation actually falls due, as YYYY-MM-DD, computed by the COUNTING RULES given above and by nothing else. Null — and only null — when the period runs from a triggering event the Order does not date. Never copy party_calculated_date into this field, and never fall back to the Order date for an undated event

Return a JSON object with exactly these keys: matter_number, order_date, deadlines
`deadlines` is an ARRAY. Each element is an object with exactly these keys: paragraph, item, basis, period_days, trigger_event, trigger_event_date, stated_date, party_calculated_date, due_date
Return an empty array only if the Order sets no deadline at all.
Use null for any value the Order does not state.

SCHEDULING ORDER
----------------
Matter Number
-------------
MVC-27-64660

Order Date
----------
29 March 2027

Recorded Events
---------------
the close of fact discovery    19 May 2027
service of written discovery   not recorded

Deadlines Ordered
-----------------
1. Each party shall serve the opposition to dispositive motions within 90 days of the date of this Order.

2. Each party shall file the joint status report no later than 3 October 2027.

3. Within 15 business days after service of written discovery, the parties shall file dispositive motions.

4. The requirement that the pretrial statement be served within 14 business days after the case management conference is WITHDRAWN and sets no date here.

5. The parties shall serve the pretrial statement within 35 days after the close of fact discovery. (the parties calculate this as 24 July 2027)

6. Within 20 days of the date of this Order, the parties shall exchange the witness list.

Raw responseThe raw response

{"matter_number":"MVC-27-64660","order_date":"2027-03-29","deadlines":[{"paragraph":1,"item":"the opposition to dispositive motions","basis":"calendar_days_from_order","period_days":90,"trigger_event":null,"trigger_event_date":null,"stated_date":null,"party_calculated_date":null,"due_date":"2027-06-28"},{"paragraph":2,"item":"the joint status report","basis":"explicit_date","period_days":null,"trigger_event":null,"trigger_event_date":null,"stated_date":"2027-10-03","party_calculated_date":null,"due_date":"2027-10-03"},{"paragraph":3,"item":"dispositive motions","basis":"business_days_from_event","period_days":15,"trigger_event":"service of written discovery","trigger_event_date":null,"stated_date":null,"party_calculated_date":null,"due_date":null},{"paragraph":5,"item":"the pretrial statement","basis":"calendar_days_from_event","period_days":35,"trigger_event":"the close of fact discovery","trigger_event_date":"2027-05-19","stated_date":null,"party_calculated_date":"2027-07-24","due_date":"2027-06-23"}]}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Turn a court's scheduling order into deadlines — 52 scheduling orders. One model answered, and every answer was then graded Four different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran this run publishes no rate card this repo commits, so nothing here is what was actually paid — the real spend for this kit's build is recorded in the commit history and the shared call ledger, not on this page.

The gradersFour ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The desk-calendar floor (evals/baseline.py), the label pre-flight (evals/check_labels.py) and the scorer (evals/judge.py) are all pure code and cost $0.00 to run against any result set. The figure above is the ONE scored run's own token counts (52 orders, 138,079 input and 267,600 output) priced at Google Gemini 3 Flash's published rate — the same basis cost_per_query_usd uses. ⚠︎ IT IS NOT WHAT BUILDING THIS KIT COST. Eighteen further live calls sit outside it: a 6-order calibration that MEASURED max_tokens and caught a cap 16 pct too low ($0.1386, committed at results/eval-c000-order-dates-calibration.json), one call probing whether a 32,000-token ceiling is accepted at all, 8 prompt-split calls at max_tokens=1 ($0.0075, committed at results/tokens-p001/p002-order-dates.json), one worked example ($0.0128, committed at results/example-ORD-0018.json) and two UI screenshot calls that are not committed as result files. All 70 live calls together project to about $1.06 on this card.

Cost driversWhat actually moves the bill

• Provider-side reasoning. 244,308 of the run's 267,600 output tokens (91.3 pct) were reasoning rather than the JSON array, on a reply that is about 450 tokens of actual content. Counting a business-day period is day-by-day work and the model does it out loud. Reasoning was left at the provider's default and nothing here has measured what turning it off would cost in accuracy.
• The fixed prefix — system prompt (752 tokens), counting rulebook (904) and field schema (771) — is 2,427 of 2,644 tokens on the worked example, 91.8 pct, and is byte-identical on all 52 calls. The order itself is 217 tokens. The bill is the rulebook and the instructions, not the document.
• Output length scales with the number of obligations, not with the length of the order: an order with six deadlines is six lots of counting, and each is its own reasoning chain.

Your volumeWhat it costs at your volume

Linear in orders: each call is independent, carries the same fixed prefix and shares nothing with its neighbours, so 520 orders cost ten times 52. Nothing amortises — there is no index and no cache. Wall clock does NOT scale linearly with the same worker pool: at EVAL_WORKERS=12 the 52 orders took 233 seconds against a 40.6-second median call, so 520 orders is about 39 minutes at the same concurrency and the knob is the only thing that changes that. ⚠︎ AND THE RULEBOOK DOES NOT SCALE LINEARLY, IT SCALES WORSE: it is sent whole in every call, so a practice covering four jurisdictions and five years of closures multiplies the input side of every request at once. At that size the holiday set has to move into code before the call, with only the relevant window sent.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 52 scheduling orders, one completion call each, on the fast tier. The calibration run, the prompt-split calls, the worked example and the UI screenshots are recorded separately in Cost.cost_of_evaluation_usd and are not projected here.

The codeEvery file, and what it is for

Six modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pysegment

cut the order into addressable sections AND into its numbered paragraphs, pure code — the paragraph is the addressable unit here, because one section holds every obligation and a span citing 'Deadlines Ordered' would prove nothing

# Cut a document into addressable sections. Pure code — no model, no network.

def sections(text):
def locate(text, value):
def span_label(secs, start):
def numbered(text):

src/select.pyselect — a swap seam

pick which sections carry each field, pure code — deadlines is mapped to the Order Date, the Recorded Events table AND the ordered paragraphs, because an obligation cannot be dated from its own paragraph alone; Court and Division are mapped by nothing and never sent

You change it to: map fields to your own order's headings; unmatched falls back to the whole document

# Pick which sections plausibly carry each field. Pure code -- the last deterministic step before

SECTION_HINTS = {
def for_field(secs, field):
def plan(secs, fields):

src/calendar_rules.pycalendar_rules

the counting rulebook, loaded from data/rulebook.json — business days, the invented holiday set, whether the trigger day counts, which bases roll and which never do. Data, not code, so a forker can open it and disagree with it

# THE COUNTING RULEBOOK, loaded from data/rulebook.json. Pure code, no model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RULEBOOK_PATH = os.path.join(HERE, "data", "rulebook.json")
BASES = ("explicit_date", "calendar_days_from_order", "calendar_days_from_event",
FROM_EVENT = ("calendar_days_from_event", "business_days_from_event")
FROM_ORDER = ("calendar_days_from_order", "business_days_from_order")
BUSINESS = ("business_days_from_order", "business_days_from_event")
CALENDAR = ("calendar_days_from_order", "calendar_days_from_event")
def load(path=RULEBOOK_PATH):
R = load()

src/prompt.pyprompt

assemble one call for the whole order, with the rulebook RENDERED into it from data/rulebook.json rather than restated in prose — a holiday list the model has never been shown is not a holiday list

# Assemble the scheduling-order prompt. One prompt per order, every obligation in it.

SYSTEM = (
def rulebook_block(r=None):
RULEBOOK_TEXT = rulebook_block()
def field_schema(fields):
def _shape(fields):
def build(doc_text, secs, fields, selector):
def parse(raw, fields):

src/extract.pyextract — a swap seam

the AI layer, one provider one key — plus two pure-code passes downstream: the date recomputed from the reply's own values, and the business-condition flag that says an obligation cannot be dated from the four corners of the order

You change it to: the undatable rule itself — this kit ships two values (the basis runs from an event, and that event has no recorded date) and a real desk also knows what the file says, what the parties have agreed and what the clerk's record shows. It is deliberately NOT the same function as the date arithmetic, so changing WHAT GETS CHASED does not change WHAT A DEADLINE IS

# Extract one scheduling order's deadlines: segment, select, prompt, one model call, then two

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 32000
SCORED_SUBFIELDS = ("item", "basis", "period_days", "trigger_event", "trigger_event_date",
SPANNABLE_SUBFIELDS = ("item", "trigger_event")
def load_fields():
def subfields(fields):
def load_doc(order_id):

evals/judge.pyjudge

score obligation discovery, field extraction, the DATE and the undatable flag separately — plus found_but_misdated, the number that separates a reading failure from a counting one, all pure code

# Score a scheduling-order run. PURE CODE -- gold is exact and every answer is one value, so `==`

ORDER_FIELDS = ("matter_number", "order_date")
NON_SPANNABLE = tuple(f for f in SCORED_SUBFIELDS if f not in SPANNABLE_SUBFIELDS)
def norm(v):
def equal(got, want):
def _cell(got, want):
def score(fields, records, golds):
def _matrix(rows, positive):
def _pct(a, b):
def score_dates(records, golds, cells):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 2655 input and 5146 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's scheduling orders are entirely synthetic (tools/build_corpus.py, seed 20260822): no real court, judge, case, docket number, party, counsel or jurisdiction exists in the corpus, and nothing was fetched from anywhere. The matter numbers are an invented shape and the obligations are generic docketing nouns. The only outbound traffic the kit makes is one chat-completion request per order, carrying the mapped sections of one order plus the fixed prompt and the shipped rulebook — the Court and Division and Courtroom sections are mapped by nothing and never leave the machine. Nothing is written outside the kit directory, there is no database, no auth and no multi-tenancy, and the local UI binds 127.0.0.1 only.

Read from the shared .env or the real environment only, never committed, never logged and never rendered. .env is gitignored from the first commit. On an adapter error the app redacts the key and base URL out of the message before returning it, so a misconfigured endpoint cannot echo a credential into a browser — which matters here because this kit deliberately surfaces transport errors that carry a URL.

The experimentWe did not attack it — and three of four boundaries hold

The three boundaries that hold were confirmed by reading the code, not by a run: no code path files, serves, dockets, waives or extends anything; an undated triggering event returns null with the missing event named, and there is no fallback branch to reach; and the key and base URL are redacted out of any error before it is returned. The fourth is open. An indirect prompt injection needs a document an outside party authored, and on this shape that is the whole document — a proposed order is normally drafted by a party and adopted by the court, which makes it the obvious place to attack and the reason the gap is named rather than glossed. Confirmed by reading the code, not by a run, on 2026-08-22 — no attack run was fired; see redteam.why.

The first three boundaries hold, confirmed by reading the code rather than by an attack run. The fourth is open and is named as open — the order text is authored outside this pipeline, which makes it the obvious place to attack, and nothing here has tried it.

The result0 attack trials, and three of four boundaries checked here hold on evidence. The one that does not is the one this kit's own shape invites: a proposed order is drafted outside the pipeline, and whether an instruction inside one could move a computed court deadline is unmeasured.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260822), so no text in it came from an outside party and there was nothing adversarial to resist. The planted ambiguity is ordinary drafting that misleads: a parenthetical carrying somebody else's arithmetic, wrong on 40 obligations, and 30 paragraphs that name an item, a number and a unit and then withdraw the requirement. That measures whether a model counts rather than copies. It does not measure whether a model obeys an instruction hidden in the same document. Those are different failures and only one of them is measured here.

HonestyWhat this does not prove

• Whether an instruction embedded in an ordered paragraph could move due_date. No attack run exists. The pure-code recomputation would disagree with a moved date, and the consistency diagnostic counts exactly that — but nobody has fired the test, so how often it would catch it is unknown.
• Whether the kit behaves safely against a hostile provider — a response body crafted to break the JSON extraction in src/prompt.py::parse, or to return an enormous payload. parse() fails closed to an empty dict (and fails closed SEPARATELY on the deadlines array, so a reply whose top level parses and whose array is a string is recorded as a failed document rather than as an order that sets no deadlines), and nothing beyond that was tested.
• Whether a real order would carry anything sensitive this kit mishandles. The corpus has no personal data by construction; a real order names parties, counsel and a live matter, and none of that path is exercised.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

`undatable` fires when the period runs from a triggering event AND the Order's own Recorded Events table gives that event no date — basis in (calendar_days_from_event, business_days_from_event) and trigger_event_date is null. Both values come out of the same reply; the rule is run afterwards in pure code, over whatever the model returned, and never over gold. A reply missing either value returns None rather than False — an unknown is not a pass. An explicit_date row with no date returns True, because a row nobody can date belongs on a chase list whichever way it got there.

src/extract.py::compute(), called by extract() on every obligation, by the local app on every click, and by evals/baseline.py on the free floor's own output so the two are routed by identical code. The date arithmetic it sits beside lives in a SEPARATE module, src/calendar_rules.py::compute(), which is also what the corpus generator wrote gold with and what src/prompt.py states to the model — one definition, three readers.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT a docket entry, a filing, a service, a waiver or an extension. Nothing here puts a date on any calendar, sends anything anywhere, or writes to any system, and due_date is arithmetic with its working attached rather than a deadline anybody is bound by. Nothing in this kit is legal advice.
• It is NOT a check on whether the extracted values are right. If the model misreads a period or an event date and then counts correctly from the misreading, the row is dated confidently on a wrong fact and this rule cannot tell. The consistency diagnostic in evals/judge.py is the closest thing to that check, it is reported separately, and it is blind to exactly the same case.
• It is NOT a docketing desk's real policy. 'The event has no recorded date' is this kit's own simplification, invented for this corpus. A real desk also knows what the file says, what the parties have agreed, what the clerk's record shows and which of those outranks the others.
• It is NOT a judgement about the rulebook. The flag reads the basis the rulebook defines; if the rulebook is wrong, the flag is confidently wrong with it. MV-CR-1 is invented and is not an authority.

WatchedWhat is watched, and why that one

The kit measures 13 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 6 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (documents 52, extraction_cells 1924, failures 0, refusal_cells 0) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

4 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 2 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

extraction · with the model in the path — 3 runs. Columns here are only ever compared with each other.

not a time series No two of these 3 runs measured the same system — they differ on documents, extraction_cells, provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

extraction · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 7 chips that all say so.

DeviationsWhat deviated

0 breaches across 4 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the flag's evidence on any change to compute() in src/extract.py, on any change to data/rulebook.json (which moves the bases the flag reads), on any change to data/fields.json's allowed values for basis, and on any corpus regeneration. Changing WHAT GETS CHASED is a policy change and must be re-scored, even though it never changes what a deadline is.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt is empty on purpose: the corpus is generated in-process, the rulebook is a JSON file read with json, the date arithmetic is datetime and a loop, the provider is reached over urllib, and the UI is one HTML file with no build step. A forker runs this on whichever key they already hold, with one clone and no install.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph. The pipeline has one path per order — segment, select, prompt, one call, parse, recompute, flag — with no branch, no loop and no agent. The only fan-out is inside one reply, where an order carries several obligations and the arithmetic runs once per row in pure code.

The other sideWhat a framework costs you

• Everything is hand-rolled, so everything is yours to maintain — the JSON extraction from a fenced reply, the backoff policy, two section regexes, the .env reader and the date arithmetic itself. The last of those is the one with teeth: a business-day counter is four lines and every one of them is a place to be wrong by a day.
• No framework means no framework's ecosystem — no tracing, no eval harness beyond the one in evals/, no prompt registry, no schema validation library. What ships is what is in the repository.
• The provider abstraction covers exactly two wire formats. A third provider is one function and one dict entry, and until somebody writes it the kit runs on two shapes.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-order-dates on the fast tier, 2026-08-22. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 3 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

2 runs not plotted. b000-order-dates-desk-calendar, t000-order-dates-stub recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — each document goes whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT — this repository's own licence. Every court name, matter number, event and obligation is invented; no real court, judge, case, docket number, party or jurisdiction is named anywhere. The counting rulebook this kit ships (data/rulebook.json, MV-CR-1) is INVENTED and is not an authority: it reproduces no procedural code, court rule, local rule, standing order or published holiday schedule, and every court holiday on its list — name and date — was made up for this kit and chosen deliberately not to coincide with any real one. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineFinding the obligations, and not inventing any

Of the 338 numbered paragraphs across 52 orders, did the run return a row for every one of the 260 that SETS a deadline, and for none of the 78 that does not? Recall and precision are reported separately and an INVENTED obligation is named on its own: on a docketing queue it is a diary entry nobody owes and somebody has to clear it, which is a different cost from a deadline that was missed.

MethodHow the test was run

evals/judge.py::score_dates, in-process, no key and no model.

Every grader on these pages scored the same 52 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's own record of which paragraph numbers carry an obligation, which carry ordinary noise and which are deadline-SHAPED and set nothing; evals/check_labels.py asserts no paragraph is labelled both ways and that every labelled paragraph exists in its order.

How tight can the band be? No threshold — a paragraph either carries a row or it does not.

Cadence: Re-run on any change to tools/build_corpus.py's paragraph templates, to the negation list in evals/baseline.py, or to rule 2 of the system prompt.

The decisionWhen to reach for it

In one linePer-field exact match, light normalisation

Does the run's value for each structured field match gold, after trimming whitespace and punctuation? Seven fields per obligation plus two per order — 1,924 cells. Four of the seven are legitimately null on most rows (period_days on a stated date, trigger_event and trigger_event_date where nothing runs from an event, stated_date where a period is counted, party_calculated_date where no party wrote one), so a null that matches gold is a hit and never a miss by default. due_date is NOT graded here — it is arithmetic and it has its own grader.

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model. The same function evals/run.py calls after every call, and the same logic the free floor is scored by — a floor and a real run scored by two different scorers cannot be compared honestly.

Every grader on these pages scored the same 52 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, read back off the same values the paragraph states; evals/check_labels.py asserts every non-nullable subfield is populated on every one of the 260 obligations, and that every item and trigger_event is present in its OWN paragraph rather than anywhere in the order, before any run is allowed to spend.

How tight can the band be? No tolerance band — exact match after trimming whitespace and punctuation, with one coercion: a digit string is compared as the integer it is.

Cadence: Re-score on any change to data/fields.json, src/prompt.py or tools/build_corpus.py. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineThe calendar date, and found-but-misdated

Did due_date land on the exact calendar date the shipped rulebook produces — including null, and only null, where the triggering event has no recorded date? Scored on its own, never averaged with the reading. ⚑ ITS HEADLINE IS found_but_misdated: obligations where ALL SEVEN structured values are exact and the date is still wrong. That is the only figure that separates a reading problem from a counting problem. Beside it sits false_dated — an obligation nothing on the order dates, given a confident date anyway.

MethodHow the test was run

evals/judge.py::score_dates, in-process, no key and no model. It is handed score()'s own cell list so found_but_misdated is computed from the SAME comparisons the extraction grade used rather than from a second, subtly different one.

Every grader on these pages scored the same 52 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: Gold's due_date is re-derived inside the grader by the same computation the kit publishes — src/calendar_rules.py::compute(), reading data/rulebook.json — so the truth this grades against can never be a separately-typed label that drifted from the rule.

How tight can the band be? No threshold — a date is one ISO string and null is a distinct answer, never folded into 'did not answer'.

Cadence: Re-run whenever data/rulebook.json or src/calendar_rules.py changes, whenever the corpus is regenerated, and on any provider or model change.

The decisionWhen to reach for it

In one lineThe undatable flag against the same rule run over gold

Does the pure-code flag — the period runs from an event AND that event has no recorded date — land on the same obligations it would land on if every field had been read perfectly? It is a BUSINESS CONDITION, so unlike a self-consistency check it genuinely needs labels, and saying so is half of what makes the number believable.

MethodHow the test was run

evals/judge.py::score_dates, in-process, no key and no model.

Every grader on these pages scored the same 52 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/extract.py::compute(), the same function the run uses, applied to GOLD's own basis, trigger_event and trigger_event_date. One rule, two readers, so a change to the rule moves both sides together and the grader cannot silently grade an old policy. evals/check_labels.py asserts the two agree on all 260 before any run may spend.

How tight can the band be? No threshold — two values and a comparison.

Cadence: Re-run whenever compute() changes. Changing WHAT GETS CHASED is a policy change and must be re-scored, even though it never changes what a deadline is.

The decisionWhen to reach for it