The business caseThe problem this solves

An application form states what it needs -- a field, a type, a format rule, whether it is mandatory -- and the package that arrives with it states most of that somewhere across five documents. The schema does not say WHICH document, and that is not an omission: a schema says what a field is, and which paper answers it is the reading. Most of the job is transcription. The rest is four derivations nobody wrote down (a total added across the goods lines of a schedule, a quantity added across the same lines when one part occupies two of them, a classification chosen off a code list by a stated attribute against a threshold, and a window counted between two dates with both endpoints included) -- in front of a package that prints values of the RIGHT SHAPE in the wrong place on every one of its 60 instances. The filer's manual walk of five documents against fourteen schema fields -- deciding which document answers each field, telling the applicant's registration number from the preparer's and the consignee's, telling the declared value of the goods from an amount payable that includes freight, telling the shipment window from a contract period, and adding up a schedule -- before a person checks and signs the form.

The inputThe actual licence application packages

The corpus is 60 licence application packages, 0.15 MB (txt 60). A per-field answer key for an application package is not public anywhere, for any form: an authoritative statement of which document answers which field, what each value is, which lines a derived value was added from, and which fields the package genuinely does not answer. That judgement is made inside a filing team, under time pressure, and is not published. Collecting real packages would have produced documents with no labels; labelling them by hand would have produced one person's reading measured against itself, and would have made the result a measurement of ONE filer's document conventions rather than of the reading task. So the corpus is generated with its facts injected, the key is read back off the rendered text with every span asserted to slice to the words it quotes, and a second implementation re-derives all fourteen fields. THREE NEAR-MISSES ARE STRUCTURAL AND APPEAR IN ALL 60 PACKAGES, clean ones included -- the consignee's own registration number in the applicant's shape, an amount payable that includes freight and is therefore not the declared value, and a labelled contract period that is not the shipment window -- because a package without them measures a tidier job than the real one.

APPLICATION PACKAGE  APP-0001
Form                 Foundry Illustrative Export Licence Application, Form FX-1
Submitted            2026-12-17

===== DOCUMENT 1 of 5 - TECHNICAL DESCRIPTION =====

Prepared by             Vantry Technical Services Limited
Item description        Compressor housing assembly, machined
Item family             Frames and housings
Attributes
  Material                        aluminium alloy 7075
  Maximum operating temperature   505 C
  Finish                          hard anodised
Narrative
  The assembly is supplied as a direct replacement for units already in service with the end user
  and is manufactured to the applicant's own drawing set. No modification to the installed
  configuration is proposed and no design data accompanies the consignment.

===== DOCUMENT 2 of 5 - PARTY DETAILS =====

Applicant
  Legal name              Harlow Aeronautical Systems Limited
  Registration number     GB-4344761
  Registered address      17 Calder Way, Harlow

Consignee
  Legal name              Kandara Air Services Private Limited
  Country                 India
  Registration number     IN-4162973

===== DOCUMENT 3 of 5 - END-USE STATEMENT =====

Stated end use          repair and return of avionics line-replaceable units
End user                Kandara Regional Airlines
End user status         Commercial entity
Intermediate consignee  Delvaux Freight Consolidation NV
Contract period         From 2027-01-29 to 2027-12-20
Declaration
  The applicant declares that the goods will be used only for the stated end use, will not be
  re-exported without authorisation, and that the particulars given in this package are complete
  and correct.

===== DOCUMENT 4 of 5 - VALUATION SCHEDULE =====

Currency                USD

The outcomeWhat a good result looks like

The form, filled: one row per field the schema defines, in the schema's order, each FOUND, DERIVED, NOT-FOUND or FORMAT-INVALID, each value normalised, each citation naming the document and quoting the line verbatim, and each derived value carrying at most twenty words of working. A field the package genuinely does not answer is filed as NOT-FOUND and the question goes back to the applicant before the form goes in, rather than after it comes back.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Packages whose fields sit against regular labels, the way this generator writes them — the free rules floor first, then decide
  805 of 840 field rows and 30 of 60 forms for $0.00 in 0.1 s, with no key and no network. The paid arm gets 824 and 57, so the money buys 19 field rows and 27 forms here -- real, and not what it costs on packages this regular. Measure your own labels before assuming either column transfers.
• Packages that print same-shaped values in the wrong place, or state a value inside a sentence — the paid arm, and read the direction metrics rather than the aggregate
  This is the whole of what the money bought and it is measured: planted rows 39 of 40 against 5 of 40, near-misses taken 0 of 195 against 25, invented 0 of 40 against 5, abandoned 0 of 800 against 5, wrong-document citations 0 against 10. By planted case the model files 5 of 5 forms on forwarder_registration, prior_auth_ghost_ref, split_line_no_part and unlabelled_consignee where the floor files 0 of 5 on each. The floor cannot tell whose a value is and this arm can.
• Deciding whether to buy anything at all on your own packages — run the free floor on your own corpus first
  python3 -m evals.run --run-id b000-yours --floor rules costs nothing and needs no key. If your labels are as regular as this generator's, the floor will do half the forms before you spend anything.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-31. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-31 — r001-licence-application. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — ⚠︎ NOT RE-MEASURED FOR THIS SPEC beyond the corpus rebuild -- stated from what ships in the repo. requirements.txt names no third-party package; the kit is standard-library Python 3 plus five JSON files. The whole free half runs with no key and no network: rebuild the corpus, grade the key, run the rules floor, run the stub arm and open the board. WHAT WAS RE-MEASURED THIS SESSION: tools/build_corpus.py under PYTHONHASHSEED 0, 1, 12345 and 99991, four separate clean copies, every one byte-identical to the committed corpus, key and stats file; and evals/check_labels.py, which reports KEY CLEAN over 840 rows and 1,165 spans.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. one filled form -- one package of five documents, one model call, end to end including provider-side reasoning tokens, on a shared connection. ⚠︎ NOT AN SLA: this credential is shared with sibling kits and these figures are a wall-clock observation of a single run. 60 packages were attempted in 1374.0 wall seconds (22.9 minutes) at 5 concurrent workers (evals/run.py defaults EVAL_WORKERS to 5), against 6281.5 seconds of summed call time -- a 4.57x ratio, which is how the concurrency is known rather than assumed. The slowest single call took 234286 ms. LATENCY, NOT MONEY, IS THE CONSTRAINT: $0.544132 for the 59 packages that answered is nothing and a p95 of 171.7 s a package is what decides whether a form is assembled while the applicant is still on the phone. The free rules floor answers all 60 in 0.1 s of wall clock, single-threaded, with no network at all (p50 0 ms, p95 1 ms).

Current processWhat it replaces

The filer's manual walk of five documents against fourteen schema fields -- deciding which document answers each field, telling the applicant's registration number from the preparer's and the consignee's, telling the declared value of the goods from an amount payable that includes freight, telling the shipment window from a contract period, and adding up a schedule -- before a person checks and signs the form.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one field rows on 60 filled application form is

No train/test split, because nothing is trained or tuned. Every arm sees all 60 packages and is graded on all 840 rows. The published breakdowns are by outcome class, by field and by planted case, each on its own denominator, because a single accuracy figure over 840 rows hides which of the fourteen fields an arm is losing -- and this kit has shipped exactly that failure: under its first grading of this run the 840-row aggregate read 91.1 pct while one field was at 0 of 60 and the form score at 0 of 60.

SetupWhat the setup figure measured

There is no index and no retrieval step. The schema, the control list and one package go into the prompt whole. The only in-memory reduction is src/schema.documents(), which splits the package on its document headers so a citation can be located; it runs after the reply comes back as well as before, and nothing is written and nothing persists. The 0.1 s is the free floor's whole wall clock over all 60 packages.

LicenceLicence

MIT for the kit's code and its generated corpus -- the MIT text is in the kits repository's LICENSE-PUBLIC, NOT in its root LICENSE, which reads PROPRIETARY AND CONFIDENTIAL. The kit is MIT; the repository is not. There is no third-party data here to licence and no collected or scraped material of any kind: the corpus, the schema, the control list and the answer key are all generated in process.

Bring your ownBring your own licence application packages

Drop your own packages as .txt files into data/corpus/ using the same five-document headers, add the matching rows to data/packages.json, and run the free floor first: python3 -m evals.run --run-id b000-yours --floor rules. That costs nothing and tells you whether your labels are regular enough that a model buys you anything at all. Your own form goes in data/schema.json and your own classification in data/codelist.json; neither needs a line of code to change.

⚠︎ And what stops being true when you do: ⚠︎ A REAL APPLICATION PACKAGE REACHES YOUR CONFIGURED PROVIDER VERBATIM AND WHOLE -- all five documents, every party name, every registration number, every price and every end use. Nothing here redacts, samples or summarises before the call, deliberately: pre-digesting the package is exactly where a value stated in a sentence disappears. If your packages carry material you may not send to a third party, the seam to change is src/adapters/__init__.py and a local endpoint, not the prompt.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 13,449 characters assembled, 6,076 are schemas — 45% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

The two strings actually posted for APP-0001, the first package of run r001-licence-application: src/prompt.py's render() output, system message then user message, rebuilt from the shipped module this session and matching the run record's own prompt_parts character counts exactly (1,956 / 6,076 / 1,828 / 2,671 / 918 = 13,449).

Full promptThe prompt in full

You are filling in an APPLICATION FORM from the package of supporting documents it arrived with.

Your output is the form, filled: one row per field the schema defines, in the schema's order, with
the value, where it came from, and -- where the package does not answer the field at all -- a plain
statement that it does not.

How to read the package:

- The package is several documents, each introduced by a header line. A field's answer may be in
  any of them. THE SCHEMA DOES NOT SAY WHICH. Which document answers a field is part of the job.
- More than one document can carry a value of the right SHAPE. A registration number belonging to
  the preparer of a technical note, a consignee's own registration, an amount payable that includes
  charges which are not goods, a date range that is not the shipment window, an authorisation
  reference belonging to somebody else -- each of these is real, printed, and not the answer.
  Decide by WHOSE it is and WHAT it is, never by its shape alone.
- Some fields are DERIVED: the package does not print them and you must work them out from what it
  does print. The schema says which, and how.
- A field the package genuinely does not answer is answered NOT-FOUND, with no value and no
  citation. That is a correct answer and it is not the same as guessing. Do not fill a field from a
  value that belongs to something else.
- A value the package DOES state but which does not satisfy the field's format rule is answered
  FORMAT-INVALID, with the value exactly as the package states it. The package is wrong, not you.
- CITE EVERY VALUE. Name the document and quote from it verbatim. Quote THE WHOLE LINE the value is
  printed on, without its leading indentation. Where the value is stated inside a sentence rather
  than against a label, quote that sentence. For a value derived from several lines, cite every
  line the derivation used.

Reply with JSON and nothing else, in the shape given at the end.

THE APPLICATION AND ITS FIELD SCHEMA

Form: Foundry Illustrative Export Licence Application, Form FX-1
Documents this package should contain, by id:

  technical-description    TECHNICAL DESCRIPTION
  party-details            PARTY DETAILS
  end-use-statement        END-USE STATEMENT
  valuation-schedule       VALUATION SCHEDULE
  prior-authorisations     PRIOR AUTHORISATION REFERENCES

Fields. EVERY ONE IS MANDATORY unless it says otherwise, which means a NOT-FOUND is a
reportable gap rather than a shrug.

  applicant_legal_name
    label      Applicant legal name
    type       text
    format     the name as written, in full
    mandatory  yes
    note       The party asking for the licence, exactly as the package writes it. Not the preparer of the technical description and not the freight forwarder.

  applicant_registration_no
    label      Applicant registration number
    type       code
    format     two capital letters, a hyphen, seven digits   (pattern ^[A-Z]{2}-[0-9]{7}$)
    mandatory  yes
    note       Every party in a package has one of these and they are all the same shape. The one the form wants belongs to the APPLICANT.

  consignee_legal_name
    label      Consignee legal name
    type       text
    format     the name as written, in full
    mandatory  yes
    note       The party the goods are consigned to.

  consignee_country
    label      Consignee country
    type       text
    format     the country as written
    mandatory  yes
    note       Where the consignee is.

  intermediate_consignee_name
    label      Intermediate consignee
    type       text
    format     the name as written, in full
    mandatory  yes
    note       The party that holds the consignment between the applicant and the consignee. Many consignments have none, and NOT-FOUND is the right answer for those -- but a package that names one in a sentence rather than against a label has still named one.

  end_use_summary
    label      Stated end use
    type       text
    format     the end use as the package words it
    mandatory  yes
    note       What the goods are for, in the package's own words.

  end_user_is_government
    label      End user is a government entity
    type       enum of yes/no
    format     yes or no
    mandatory  yes
    note       The package states the end user's status in words; the form wants yes or no.

  item_description
    label      Item description
    type       text
    format     the description as written
    mandatory  yes
    note       What is being shipped, from the technical paperwork rather than from the price list.

  control_classification
    label      Control classification
    type       code
    format     FICL- then a digit, a capital letter and two digits   (pattern ^FICL-[0-9][A-Z][0-9]{2}$)
    mandatory  yes
    DERIVED    classify_from_codelist
    note       Not printed anywhere in the package. Choose it from data/codelist.json: the family fixes the pair of candidate codes and one stated numeric attribute decides which of the pair applies. A package can state that attribute in its attribute block, in a sentence, or both -- and where both, the one the code list names is the one that decides.

  item_quantity
    label      Total quantity
    type       integer
    format     a whole number
    mandatory  yes
    DERIVED    sum_line_items
    note       The total quantity of GOODS across the valuation schedule. Charges that are not goods -- freight, insurance, handling -- are not quantities. A single part can occupy more than one line.

  total_declared_value_cents
    label      Total declared value
    type       money_cents
    format     an amount; recorded in integer minor units (cents) so it can be compared exactly
    mandatory  yes
    DERIVED    sum_line_items
    note       The extended value of the GOODS, added up. The schedule also prints an amount payable, which includes charges that are not goods and is therefore not the declared value of what is being exported.

  currency
    label      Currency
    type       enum of USD/EUR/GBP
    format     a three-letter code from the list
    mandatory  yes
    note       The currency the schedule is priced in.

  shipment_window_days
    label      Shipment window in days
    type       integer
    format     a whole number of days
    mandatory  yes
    DERIVED    days_between
    note       From the first shipment date to the last shipment date, COUNTING BOTH ENDPOINTS. A package carries other date pairs -- a contract period, a declaration date -- and those are not the shipment window.

  prior_authorisation_ref
    label      Prior authorisation reference
    type       code
    format     PA-, a four-digit year, a hyphen, five digits   (pattern ^PA-[0-9]{4}-[0-9]{5}$)
    mandatory  yes
    note       The authorisation already granted FOR THIS CONSIGNMENT. Many consignments have none, and NOT-FOUND is the right answer for those -- a reference of the same shape naming somebody else's authorisation is not this consignment's.

The three derivations, in words:

  sum_line_items          add the relevant column across the GOODS lines of the valuation schedule.
                          A goods line is a line of the ordered items. Charges that are not goods are not
                          part of either sum, and one part can occupy more than one line.
  classify_from_codelist  pick the code from the control list supplied below.
  days_between            count from the first date to the second, COUNTING BOTH ENDPOINTS.

The four outcomes, and when each applies:

  FOUND           the package states the value and you have copied it.
  DERIVED         the package does not state the value; you worked it out from what it does state,
                  by the derivation the schema names for that field.
  NOT-FOUND       the package does not answer this field at all. No value, no citation.
  FORMAT-INVALID  the package states a value for this field and that value does not satisfy the
                  field's format rule. Report it exactly as stated.

THE CONTROL LIST -- Foundry Illustrative Control List (FICL)

The item's FAMILY fixes a pair of candidate codes. ONE stated numeric attribute -- named by the family, in the unit the family names -- decides which of the pair applies. Nothing else in the package is part of the decision.

Families and the attribute that decides:

  Frames and housings        decided by maximum operating temperature, in C
      below 400 C -> FICL-1A01        400 C or above -> FICL-1A02
  Rotating assemblies        decided by maximum tip speed, in m/s
      below 250 m/s -> FICL-2B11        250 m/s or above -> FICL-2B12
  Signal processing units    decided by maximum sample rate, in MSPS
      below 500 MSPS -> FICL-3C21        500 MSPS or above -> FICL-3C22
  Navigation modules         decided by stated position accuracy, in m
      below 10 m -> FICL-4D32        10 m or above -> FICL-4D31
      NOTE: ⚑ THIS FAMILY RUNS THE OTHER WAY and it is the one that catches a reader skimming the pattern. Accuracy improves as the number falls, so a SMALLER number is the more tightly controlled code. Every other family here puts the higher code above the threshold.

The codes:

  FICL-1A01    Frames and housings with a maximum operating temperature below 400 C
  FICL-1A02    Frames and housings with a maximum operating temperature of 400 C or above
  FICL-2B11    Rotating assemblies with a maximum tip speed below 250 m/s
  FICL-2B12    Rotating assemblies with a maximum tip speed of 250 m/s or above
  FICL-3C21    Signal processing units with a maximum sample rate below 500 MSPS
  FICL-3C22    Signal processing units with a maximum sample rate of 500 MSPS or above
  FICL-4D31    Navigation modules with a stated position accuracy of 10 m or coarser
  FICL-4D32    Navigation modules with a stated position accuracy better than 10 m

THE PACKAGE, verbatim and whole:

APPLICATION PACKAGE  APP-0001
Form                 Foundry Illustrative Export Licence Application, Form FX-1
Submitted            2026-12-17

===== DOCUMENT 1 of 5 - TECHNICAL DESCRIPTION =====

Prepared by             Vantry Technical Services Limited
Item description        Compressor housing assembly, machined
Item family             Frames and housings
Attributes
  Material                        aluminium alloy 7075
  Maximum operating temperature   505 C
  Finish                          hard anodised
Narrative
  The assembly is supplied as a direct replacement for units already in service with the end user
  and is manufactured to the applicant's own drawing set. No modification to the installed
  configuration is proposed and no design data accompanies the consignment.

===== DOCUMENT 2 of 5 - PARTY DETAILS =====

Applicant
  Legal name              Harlow Aeronautical Systems Limited
  Registration number     GB-4344761
  Registered address      17 Calder Way, Harlow

Consignee
  Legal name              Kandara Air Services Private Limited
  Country                 India
  Registration number     IN-4162973

===== DOCUMENT 3 of 5 - END-USE STATEMENT =====

Stated end use          repair and return of avionics line-replaceable units
End user                Kandara Regional Airlines
End user status         Commercial entity
Intermediate consignee  Delvaux Freight Consolidation NV
Contract period         From 2027-01-29 to 2027-12-20
Declaration
  The applicant declares that the goods will be used only for the stated end use, will not be
  re-exported without authorisation, and that the particulars given in this package are complete
  and correct.

===== DOCUMENT 4 of 5 - VALUATION SCHEDULE =====

Currency                USD

Line   Part number   Description                           Qty     Unit value   Extended value
1      HAS-40118     Compressor housing                     58       2,972.32       172,394.56
2      HAS-40119     Retaining collar                       89       4,797.69       426,994.41
3      HAS-40120     Balance ring                           74       3,931.20       290,908.80
-      -             Freight and insurance                   -              -         2,738.11
                     Total payable                                                  893,035.88

Delivery terms
  First shipment        2027-06-11
  Last shipment         2027-08-01

===== DOCUMENT 5 of 5 - PRIOR AUTHORISATION REFERENCES =====

Prior authorisation     None. No prior authorisation has been granted for this consignment.
Enquiries               the contact named on the covering form


Reply with JSON and nothing else, exactly this shape:

{"application_ref": "<the package reference printed at the top, verbatim>",
 "fields": [
   {"field": "<the field id from the schema, exactly>",
    "outcome": "FOUND" | "DERIVED" | "NOT-FOUND" | "FORMAT-INVALID",
    "value": "<the value as a string, or null for NOT-FOUND>",
    "cite": [{"doc": "<the document id the quote is from>",
              "quote": "<verbatim from that document -- the whole line, or the whole sentence>"}],
    "working": "<for DERIVED only: at most twenty words saying how you got it. Otherwise \"\">",
    "why": "<at most fifteen words: why this value and not another>"}
 ]}

One row per field in the schema, in the schema's order, and no row for anything the schema does not
define. Amounts may be written with or without thousands separators; they are compared as amounts,
not as strings. Dates in a value are written YYYY-MM-DD.

Raw responseThe raw response

{
  "application_ref": "APP-0001",
  "fields": [
    {
      "field": "applicant_legal_name",
      "outcome": "FOUND",
      "value": "Harlow Aeronautical Systems Limited",
      "cite": [
        {
          "doc": "party-details",
          "quote": "Legal name              Harlow Aeronautical Systems Limited"
        }
      ],
      "working": "",
      "why": "Printed under Applicant; not the technical preparer or forwarder."
    },
    {
      "field": "applicant_registration_no",
      "outcome": "FOUND",
      "value": "GB-4344761",
      "cite": [
        {
          "doc": "party-details",
          "quote": "Registration number     GB-4344761"
        }
      ],
      "working": "",
      "why": "Printed under Applicant; belongs to the applicant, not consignee."
    },
    {
      "field": "consignee_legal_name",
      "outcome": "FOUND",
      "value": "Kandara Air Services Private Limited",
      "cite": [
        {
          "doc": "party-details",
          "quote": "Legal name              Kandara Air Services Private Limited"
        }
      ],
      "working": "",
      "why": "Printed under Consignee in party details."
    },
    {
      "field": "consignee_country",
      "outcome": "FOUND",
      "value": "India",
      "cite": [
        {
          "doc": "party-details",
          "quote": "Country                 India"
        }
      ],
      "working": "",
      "why": "Printed under Consignee country in party details."
    },
    {
      "field": "intermediate_consignee_name",
      "outcome": "FOUND",
      "value": "Delvaux Freight Consolidation NV",
      "cite": [
        {
          "doc": "end-use-statement",
          "quote": "Intermediate consignee  Delvaux Freight Consolidation NV"
        }
      ],
      "working": "",
      "why": "Named under Intermediate consignee in end-use statement."
    },
    {
      "field": "end_use_summary",
      "outcome": "FOUND",
      "value": "repair and return of avionics line-replaceable units",
      "cite": [
        {
          "doc": "end-use-statement",
          "quote": "Stated end use          repair and return of avionics line-replaceable units"
        }
      ],
      "working": "",
      "why": "Printed as stated end use in end-use statement."
    },
    {
      "field": "end_user_is_government",
      "outcome": "FOUND",
      "value": "no",
      "cite": [
        {
          "doc": "end-use-statement",
          "quote": "End user status         Commercial entity"
        }
      ],
      "working": "",
      "why": "Commercial entity is not a government entity."
    },
    {
      "field": "item_description",
      "outcome": "FOUND",
      "value": "Compressor housing assembly, machined",
      "cite": [
        {
          "doc": "technical-description",
          "quote": "Item description        Compressor housing assembly, machined"
        }
      ],
      "working": "",
      "why": "Technical description gives the full item name."
    },
    {
      "field": "control_classification",
      "outcome": "DERIVED",
      "value": "FICL-1A02",
      "cite": [
        {
          "doc": "technical-description",
          "quote": "Item family             Frames and housings"
        },
        {
          "doc": "technical-description",
          "quote": "Maximum operating temperature   505 C"
        }
      ],
      "working": "Frames/housings; 505 C >= 400 C threshold, so FICL-1A02.",
      "why": "505 C is 400 or above; family frames/housings."
    },
    {
      "field": "item_quantity",
      "outcome": "DERIVED",
      "value": "221",
      "cite": [
        {
          "doc": "valuation-schedule",
          "quote": "1      HAS-40118     Compressor housing                     58       2,972.32       172,394.56"
        },
        {
          "doc": "valuation-schedule",
          "quote": "2      HAS-40119     Retaining collar                       89       4,797.69       426,994.41"
        },
        {
          "doc": "valuation-schedule",
          "quote": "3      HAS-40120     Balance ring                           74       3,931.20       290,908.80"
        }
      ],
      "working": "Sum goods quantities 58 + 89 + 74 = 221.",
      "why": "Sum of goods quantities; freight line not goods."
    },
    {
      "field": "total_declared_value_cents",
      "outcome": "DERIVED",
      "value": "89029777",
      "cite": [
        {
          "doc": "valuation-schedule",
          "quote": "1      HAS-40118     Compressor housing                     58       2,972.32       172,394.56"
        },
        {
          "doc": "valuation-schedule",
          "quote": "2      HAS-40119     Retaining collar                       89       4,797.69       426,994.41"
        },
        {
          "doc": "valuation-schedule",
          "quote": "3      HAS-40120     Balance ring                           74       3,931.20       290,908.80"
        }
      ],
      "working": "Sum extended values 172,394.56 + 426,994.41 + 290,908.80 = 890,297.77.",
      "why": "Goods extended values sum; total payable excludes freight."
    },
    {
      "field": "currency",
      "outcome": "FOUND",
      "value": "USD",
      "cite": [
        {
          "doc": "valuation-schedule",
          "quote": "Currency                USD"
        }
      ],
      "working": "",
      "why": "Printed currency of valuation schedule."
    },
    {
      "field": "shipment_window_days",
      "outcome": "DERIVED",
      "value": "52",
      "cite": [
        {
          "doc": "valuation-schedule",
          "quote": "First shipment        2027-06-11"
        },
        {
          "doc": "valuation-schedule",
          "quote": "Last shipment         2027-08-01"
        }
      ],
      "working": "Inclusive days from 2027-06-11 to 2027-08-01 is 52.",
      "why": "Delivery terms are shipment window; both endpoints counted."
    },
    {
      "field": "prior_authorisation_ref",
      "outcome": "NOT-FOUND",
      "value": null,
      "cite": [],
      "working": "",
      "why": "Package expressly states no prior authorisation exists."
    }
  ]
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Fill out an export licence application, sourced line by line — 840 licence application packages drawn from 60 real licence application packages. One model answered, and every answer was then graded Three different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

No model grades anything. Every figure is exact match against data/gold.jsonl on three things graded separately: the VALUE on its stored form (== on codes, enums, counts and money -- a declared value out by one cent is wrong -- with one stated tolerance on free text only, applied identically to every arm), the OUTCOME word, and the CITATION by span overlap. A citation is credited only when recall >= 0.5 AND precision >= 0.5, both measured as character positions inside the document the arm named; recall alone would credit quoting the whole document and precision alone would credit quoting three characters of the right line. Mean recall and precision are published beside the credited count so the distribution is visible and not just the pass rate. A field the key marks NOT-FOUND carries no citation and is outside the citation denominator entirely -- 800 of the 840 rows are in it -- and src/fill.py strips any citation attached to a NOT-FOUND answer rather than letting an arm bank credit for a field it declined to fill.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Run it twiceThe same set, run again

field rows all-correct over the 840 rows —

Grading costWhat it costs

Every dollar on this page is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid. The token counts come from the provider's own usage block on each call, and 90.3 pct of the output tokens they report are provider-side reasoning the kit did not ask for.

Rates checked 2026-08-27. The provider that actually ran every call here is kept off this page per the series rule. The real spend ($0.544132 over 59 calls), its own dated card, its peak/off-peak split and its cached-input tier are all recorded per call in results/eval-r001-licence-application.json, not on this page.

Grading unitWhat the grading figure prices

Scoring is pure Python over committed JSON: 840 rows in well under a second, no key, no network. Re-scoring costs nothing, which is what makes every figure here re-derivable from the committed cache file for $0.00.

The gradersThree ways to grade

The null baseline is answering FOUND on every row, which is the majority outcome: 555 of 840 rows (66.1 pct) carry it in the key. That arm gets 66.1 pct of OUTCOMES right and 0.0 pct of values, 0 forms, and it is wrong on every one of the 40 NOT-FOUND rows -- the direction this kit says is never caught. It is quoted so nobody reads 98.2 pct outcome accuracy as the achievement; the achievement is the 285 non-FOUND rows and the 40 refusals inside them. ⚠︎ The baseline_hits column of the per-field table beside it is the FREE RULES FLOOR's per-field count, not this null arm -- the floor is the baseline a reader is being asked to beat with money.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Set limitationsWhat this set cannot show

DELIBERATELY UNBALANCED AND THE IMBALANCE IS DECLARED. 25 of 60 packages are clean and 35 carry exactly one of 7 planted cases, 5 each. By outcome the key is FOUND 555, DERIVED 240, NOT-FOUND 40, FORMAT-INVALID 5 -- so two thirds of the rows are the easiest kind and 45 rows carry the two rarest outcomes.

A single accuracy figure over all 840 rows FLATTERS EVERY ARM and hides which field it is losing. It hid nothing on the free floor, whose 35 misses spread across 6 fields. ⚠︎ IT VERY NEARLY HID EVERYTHING ABOUT THE PAID ARM, AND THIS KIT SHIPPED THE PROOF: under the pre-repair money rule the same replies reported 91.1 pct of field rows -- an unremarkable-looking number that was thirteen perfect fields and one field at 0 of 60 -- while the form score, on the same rows, was 0 of 60. The field aggregate is dominated by the ten stated fields (600 of 840 rows) and would barely have moved if a second derived field had gone to zero as well. Every rate in this lens is published on its own denominator for that reason.

Building and labelling the whole set cost nothing -- it is generated from a seed, and the key is computed rather than typed. Grading it costs nothing. The only money in this kit is the 60 calls of the paid arm. What has NOT been built is a case that separates the two ERROR DIRECTIONS on the paid arm. It scores 0 invented and 0 abandoned, so this corpus cannot say what it would take to make it invent one -- and 'zero' on a corpus that never pushed is a corpus result, not a kit result. Nor is there a case that separates an arm's UNIT CONVENTION from its arithmetic: nothing in the corpus asks for the same quantity in two units, which is exactly the axis this kit's own scorer was wrong on for a day with every gate green.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-27. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingGrading is free; the bill is the 59 calls that answered.

The scorer, the label gate, the corpus generator, the rules floor and the stub arm are all pure Python over committed files -- no key, no network, no bill. Re-scoring the committed cache makes no call at all, and that is not a claim: the model figures on this page were produced that way on 2026-08-31 at $0.00 with 0 calls re-fired. The only money this kit has ever spent is run r001-licence-application: $0.544132 actually paid at the running provider's off-peak tariff over the 59 calls that returned, which is recorded per call in the result file and is not the figure on this page. The figure on this page is the same measured tokens priced on the estate's shared projection card, extended to a full 60-package pass.

Cost driversWhat actually moves the bill

• OUTPUT TOKENS, OVERWHELMINGLY. 13598.0 per call against 3267.2 in, and 90.3 pct of them (724479 of 802281) are provider-side reasoning left at the tier's default and re-rolled per call. On the projection card the output side is 96.1 pct of the bill. The visible answer is 14 rows.
• THE STABLE PREFIX IS THE SAVING, NOT THE COST. The role, the schema and the control list are 9,860 of the prompt's 13,449 characters -- 73.3 pct identical on every call -- and the provider that ran this billed 130432 of 192766 input tokens at its cached rate (67.7 pct). The projection card has no cache tier and prices those tokens at full rate; on this kit that barely matters, because input is only 3.9 pct of the projected bill.
• THE PACKAGE ITSELF IS 19.9 PCT OF THE PROMPT. The bill is the schema, the control list and the reasoning, not the document -- so a longer package moves the cost hardly at all, and a longer FORM moves every call.

Your volumeWhat it costs at your volume

Linear in packages and flat in everything else. Ten times the packages is ten times the calls and ten times the bill: 600 packages is about $5.44 at the tariff actually paid, or $25.46 on the projection card. Nothing here indexes, caches between packages or amortises -- the only sub-linear part is the provider's own prefix cache, which already covers 67.7 pct of the input and input is a small share of the bill. Latency scales with worker count, not with corpus size: 60 packages took 22.9 minutes at 5 workers.

Model choiceWhy this model

It is the tier the estate runs, so this figure compares with every sibling kit's. It is not a recommendation and this run is not an argument for it in general: it beats the free floor here on every published aggregate, but the free floor still files 30 of 60 forms for $0.00 on a corpus written to labels a regex can reach, and no second model was called against it.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

the 59 calls of the 60-package scored run that returned (r001-licence-application, the fast tier). One call died in transport and run.py returns before accumulating tokens on an error, so the token totals above cover 59 and the projected pass below is extended to a full 60.

The codeEvery file, and what it is for

10 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/prompt.pyprompt assembly

Five parts in a fixed order and one system message plus one user message: the role and the rules of reading (1,956 chars), data/schema.json rendered field by field (6,076), the control list rendered (1,828), THE PACKAGE VERBATIM AND WHOLE, all five documents (2,671 for APP-0001), and the JSON reply shape (918) -- 13,449 characters, matching the run record's own prompt_parts exactly. It asserts at import that every field, every control code and every outcome word appears in the rendered text; scoring an arm on a vocabulary it was never given measures the prompt. The package is never pre-parsed, because a pre-parsed field summary is where a value stated in a SENTENCE quietly disappears before the model sees it. ⚑ AND ONE RENDERED LINE IS WHAT THIS RUN'S DEFECT TURNED ON, THOUGH THE LINE ITSELF WAS NEVER WRONG: the money field's format line says recorded in integer minor units (cents) so it can be compared exactly, the model obeyed it on all 59 answered packages, and src/schema.py::parse then read the reply as a major-unit amount and multiplied by a hundred. The sentence is now exactly true of the parser and was DELIBERATELY LEFT ALONE -- it is rendered into the prompt, and on a fully cached resume evals/run.py rebuilds prompt_parts from the CURRENT src/prompt.py, so editing it would silently rewrite the recorded prompt of a run fired with the old one.

# The assembled prompt, in one place, in the order it is sent.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
SYSTEM = """You are filling in an APPLICATION FORM from the package of supporting documents it arrived with.
OUTCOME_TEXT = """The four outcomes, and when each applies:
SCHEMA_REPLY = """Reply with JSON and nothing else, exactly this shape:
def _schema_text():
def _codelist_text():
SCHEMA_TEXT = _schema_text()
CODELIST_TEXT = _codelist_text()
def build(package_text, pkg=None):

src/extract.pythe model call

One call per package, the only place a provider is reached. Parses the JSON reply, hands it to the pure-code station and returns the filled form with the provider's own usage block attached. max_tokens is 32,000; a reply cut off at the ceiling would be recorded with at_ceiling and stay inside the published denominator. 60 calls were attempted and 59 answered: all 59 finished with stop, the largest reply drew 23,945 tokens (74.8 pct), and the sixtieth died in transport ([Errno 54] Connection reset by peer, at_ceiling false).

# One package in, one filled application out. The only place a model is called.

MAX_TOKENS = 32000
THINKING = None
def parse_reply(text):
def extract(cfg, package_text, pkg=None, complete_fn=None, max_tokens=None):

src/adapters/__init__.pythe adapters — a swap seam

Raw HTTP over urllib to any OpenAI-compatible provider or Anthropic -- no vendor SDK, so requirements.txt pulls nothing and a forker runs this on whichever key they already hold. The socket timeout is 1,200 s, recorded in the run file. A provider returning no token counts cannot be published here, because the LLM lens prints them and the Cost lens prices them.

You change it to: PROVIDER + BASE_URL + MODEL in .env -- one line, then the same run again. Configuration is at the REPO ROOT and inherited by every kit; a kit-local .env is an override merged key by key. A new adapter must return token counts.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
TIMEOUT_S = 1200
TRANSPORT_RETRIES = 1
def _post(url, headers, payload, timeout=TIMEOUT_S):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}

src/fill.pythe pure-code station

NORMALISE, VALIDATE, LOCATE -- identical for the model, the floor and the stub, which is what makes the three columns comparable at all. It normalises the value into the form the field is compared in (delegating every parse to src/schema.py), validates it against the field's own format rule (a located value the form cannot accept becomes FORMAT-INVALID, a third outcome and not a wrong value), and locates every citation: THE ARM IS NEVER ASKED FOR CHARACTER OFFSETS, it names a document and quotes verbatim and this file finds the quote inside that document. A field the arm did not answer at all becomes a MISSING row rather than disappearing, so a silently absent field is a failure and not a smaller denominator -- which is exactly how the dropped package's 14 rows are counted. ⚠︎ IT IS THE CALLER, NOT THE SITE, OF THIS RUN'S DEFECT: the money parser lives in src/schema.py::parse and this file calls it. Attributing the collision here, as this spec did before 2026-08-31, sends a reader to the wrong file.

# THE PURE-CODE STATION EVERY ARM GOES THROUGH. No model, no network, identical for all of them.

MISSING = "MISSING"
def settle(reply, text):
def _cites(raw, text, docs):
def positions(cites):
def gold_positions(cites):

src/schema.pythe schema and its derivations

data/schema.json and data/codelist.json as code: parse() per type, format_ok() per field, same() for the one stated tolerance (free text, case-folded and whitespace-collapsed -- codes, enums, counts and money are compared with == and a declared value out by one cent is wrong), display() as the single place money comes back out of minor units, and the three derivations. Both JSON files are read by the prompt AND by every code path, so they cannot drift from each other. ⚑ ITS money_cents BRANCH IS THE ONE SITE THIS RUN'S DEFECT LIVED IN, AND IT IS FIXED. It multiplied every arm's answer by a hundred, reading 89029777 -- an arm doing exactly what the rendered format note asked for -- as 89,029,777 dollars. The rule now: A FRACTION MEANS MAJOR UNITS, A BARE INTEGER MEANS MINOR UNITS, because minor units are whole numbers by definition, so nothing is inferred from magnitude and nothing is ambiguous. '89029777' and '890297.77' and 'USD 890,297.77' all store 89029777, and S.display(89029777) -> '890297.77' -> S.parse round-trips. It was the site OUT OF STEP, not the rule: evals/scoring._near_eq and evals/check_labels.norm already read an integer money value as minor units and neither was changed.

# The application's own field schema, the package splitter, and the pure-code stations every arm

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
SCHEMA = json.load(open(os.path.join(DATA, "schema.json"), encoding="utf-8"))
CODELIST = json.load(open(os.path.join(DATA, "codelist.json"), encoding="utf-8"))
FIELDS = SCHEMA["fields"]
FIELD_IDS = [f["field"] for f in FIELDS]
BY_FIELD = {f["field"]: f for f in FIELDS}
FAMILIES = {f["family"]: f for f in CODELIST["families"]}
CODES = {e["code"]: e["text"] for e in CODELIST["entries"]}

src/floor.pythe free floor — a swap seam

What a filing team would script with a weekend and no model: label anchors, negative openers, the schedule rule (a goods line carries a part number) and a date fallback, then all three derivations in the same code the key uses -- handed to the SAME src/fill.py station and graded by the SAME scorer against the SAME key. Everything the paid arm gets for free, this arm also gets. It files 30 of 60 forms and gets 805 of 840 field rows, and all 35 of its misses are values the package really prints somewhere else: 25 near-misses taken, 5 invented, 5 abandoned. Nothing it gets wrong is arithmetic -- derived_wrong and value_wrong are both 0. ⚑ AND IT IS WHY THE MONEY DEFECT READ AS A BROKEN ARM RATHER THAN A BROKEN SCORER: this file hands in S.display()'s decimal form, so it never met the branch that punished a bare integer, and it scored 55 of 60 on the field the model scored 0 of 60 on. An arm that obeys a format note and an arm that ignores it are not interchangeable evidence about a parser.

You change it to: The label anchors, the negative openers, the schedule rule and the date fallback. Widen these and the free column moves; this is the file a forker should edit before buying anything, because on this corpus it already files half the forms.

# THE FREE FLOOR -- label-anchored field lifting plus the arithmetic, in pure code. No key, no

NEGATIVE_OPENERS = ("none", "no ", "nil", "not applicable", "n/a", "not stated", "not provided")
LABEL_RE_CACHE = {}
def _label_lines(text, docs, label):
def _first(text, docs, label, nth=0):
def _absent(value):
def _code_token(field, value):
def _cite(doc, line):
ROW_RE = re.compile(r"^\s*(\d+)\s+([A-Z]{2,4}-\d{3,6})\s+(.+?)\s{2,}(\d+)\s+([\d,]+\.\d{2})"
DATE_PAIR_RE = re.compile(r"^.*?(\d{4}-\d{2}-\d{2}).*?(\d{4}-\d{2}-\d{2}).*$", re.M)

evals/scoring.pythe scorer — a swap seam

THE UNIT IS THE FIELD AND THE APPLICATION IS A SECOND UNIT ON ITS OWN DENOMINATOR. Rows match by field id, never by position. Value, outcome and citation are graded separately and then together, and the two error directions -- invented (denominator: the 40 NOT-FOUND rows) and abandoned (denominator: the 800 answerable rows) -- are never averaged, because one is caught by the office that receives the form and the other is not caught at all. Every miss files into exactly one of eight taxonomy buckets in a fixed order.

You change it to: The graders, the citation rule (recall and precision both >= 0.5) and the eight-bucket taxonomy. invented and abandoned are scored on their own denominators and never averaged, which is the only reason the floor's 5 invented rows are visible at all.

# Grade one arm against the answer key. Deterministic -- no model judges anything here.

OUTCOMES = ("FOUND", "DERIVED", "NOT-FOUND", "FORMAT-INVALID")
TAXONOMY = ("field_missing", "invented_value", "abandoned_field", "near_miss_taken",
CITE_MIN_RECALL = 0.5
CITE_MIN_PRECISION = 0.5
def _pct(n, d):
def cite_credit(arm_cites, gold_cites):
def _docs_of(cites):
def _bucket(g, m, value_ok, outcome_ok, cite_ok):
def _near_eq(field, got, near):

evals/check_labels.pythe label gate

Grades the ANSWER KEY with an implementation that imports nothing from src/: its own document splitter, its own label reader, its own money parser, its own date arithmetic, its own reading of the control list, and a DIFFERENT rule for the schedule -- the floor says a goods line carries a part number, the checker says it has a whole number in the quantity column and an amount in the extended-value column, so a line continued without a part number is goods to one and not to the other. It reports KEY CLEAN over 840 field rows and 1,165 re-sliced citation spans. It is red-proven rather than assumed: four seeded defects in gold.jsonl make it exit non-zero and name all four by row.

# Grade the ANSWER KEY itself, with a second implementation that does not import the kit. Free.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
SCHEMA = json.load(open(os.path.join(DATA, "schema.json"), encoding="utf-8"))
CODELIST = json.load(open(os.path.join(DATA, "codelist.json"), encoding="utf-8"))
FIELDS = [f["field"] for f in SCHEMA["fields"]]
SPEC = {f["field"]: f for f in SCHEMA["fields"]}
FAM = {f["family"]: f for f in CODELIST["families"]}
CODES = {e["code"] for e in CODELIST["entries"]}
HDR = re.compile(r"^=====\s+DOCUMENT\s+(\d+)\s+of\s+(\d+)\s+-\s+(.+?)\s+=====\s*$", re.M)

src/app.pythe local board

http.server, hand-written HTML and one JS file on 127.0.0.1:9227. /api/docs, /api/doc, /api/floor, /api/corpus, /api/prompt and /api/recorded need no key at all; only /api/fill calls a provider, and with no API_KEY it returns 200 saying so rather than failing. RECORDED_RUN defaults to r001-licence-application, so the replay route now serves the scored run -- the README's statement that the replay button is disabled describes the state before that run landed.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
CORPUS = os.path.join(HERE, "data", "corpus")
PORT = int(os.environ.get("PORT", "9227"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r001-licence-application")
FLOOR_RUN = os.environ.get("FLOOR_RUN", "b000-licence-application-rules")
def packages():
def golds():
def load_doc(did):

tools/build_corpus.pythe corpus generator

The 60 packages, the 840 keyed field rows and the 1,165 citation spans, from SEED = 20260831. The facts are injected first and the documents rendered afterwards, so there is no path where the text says one thing and the key says another; every planted case is asserted before the corpus ships -- a threshold_prose package whose nominal figure lands on the same control code would measure nothing and would otherwise have shipped silently. Re-run under four different PYTHONHASHSEED values this session, it reproduces all 60 packages, the key and the stats file byte for byte.

# Generate the application packages, their metadata and the answer key. Deterministic, free.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
CORPUS = os.path.join(DATA, "corpus")
SEED = 20260831
DOCS = 60
DATASET = "licence-application-v1-60packages"
CASES = ["forwarder_registration", "split_line_no_part", "prior_auth_ghost_ref",
PER_CASE = 5
CLEAN = DOCS - len(CASES) * PER_CASE

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 3267 input and 13598 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

⚠︎ WRITTEN, FORCED BY DESIGN, AND NOT FIRED. A supporting package arrives from outside — from the applicant, their preparer, their forwarder — as files, and it reaches the model VERBATIM AND WHOLE, because pre-digesting it is exactly where a value stated in a sentence disappears. Anybody who can put a document in the package can put a sentence in it addressed to whatever reads it, and such a sentence is indistinguishable in form from the filing notes real packages carry. Nothing here filters the input and a filter would be the wrong fix. evals/injection.py is written to force exactly that and HAS NEVER BEEN RUN: there is no x001 result file and no suppression rate anywhere in this kit.

API_KEY is read from the shared <code>&lt;repo&gt;/.env</code>, this kit's own .env as an override, or the real environment, which wins over both. Both .env paths are gitignored from the first commit. The key leaves only inside the Authorization header. Every route on the local board except /api/fill needs no key at all, and /api/fill with no key returns 200 saying so rather than failing.

The experimentWe attacked it — two gates

An indirect prompt injection has to clear two gates, and they behave differently. Both were run for real on not_measured.

Both gates are unfired as adversarial probes and neither should be read as a zero. The kit's structural defence is that everything after the reading is code: src/fill.py normalises, validates against the field's own format rule and locates every quote inside the document the arm named, so a citation that does not locate earns nothing and one that locates in a different document is recorded as exactly that. That is verifiable by reading the file and it is NOT a measurement.

The resultNot measured — the probe is written and unfired. No suppression rate exists for this kit and none is published.

There is nothing to detail: no trial was fired. What is ready is the harness — evals/injection.py appends one sentence to the technical description, pairs each injected reply against the scored run's own cached answer for the same package, and excludes any package the control itself got wrong on that field, so a control that was already wrong cannot be counted as a suppression.

HonestyWhat this does not prove

• ANY SUPPRESSION RATE. The probe is written and unrun. There is no number, and a zero must not be read into an absence.
• WHETHER A DIFFERENT SENTENCE WOULD WORK BETTER. One sentence is written into evals/injection.py and it was chosen for the two fields an application is priced on. Nothing here surveys the space.
• WHETHER THE CITATION REQUIREMENT IS A DEFENCE. It is a structural property — src/fill.py locates every quote inside the document the arm named — and it has never been tested against an arm trying to defeat it.
• ANYTHING ABOUT A REAL PACKAGE. Every document here is generated. A forged package would not have to keep its planted value locatable, which is the assumption the whole citation apparatus rests on.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Never file anything, submit anything, or decide whether a consignment may move. Never advise whether a licence is required, whether an item is controlled, or whether a party may receive it -- the control list here is invented for the kit and is an authority on nothing. Produce the form a filer checks and signs, with every value pointing at the line it came from and every unanswered field named.

Stated on the board, in the README, in data/SOURCES.md and in src/app.py's docstring, and enforced structurally by the absence of any write path: the kit has no submit route, no outbound call but the one completion, and nothing that writes anywhere but results/.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• This is ABSENCE OF A WRITE PATH plus a prompt rule, not a runtime enforcement layer. The only real enforcement is src/fill.py: it re-derives nothing, but it does normalise, validate against the field's own format rule and locate every citation, so a quote that does not locate earns no credit.
• ⚠︎ THERE IS NO INJECTION RESULT HERE AT ALL. evals/injection.py is written and unfired; no sentence has been tested against this kit and no suppression rate exists.
• It is NOT a check on the READING. Nothing re-derives which document answers a field, and the reading is where every unfixable error in this kit lives.
• ⚠︎ AND IT IS STILL NOT A CHECK ON UNITS. Nothing compares the arm's stated units against the schema's, or data/schema.json's format_note against src/schema.py::parse. That is the failure this kit already paid for: 59 rows exactly x100, every citation credited, every why correct, and no guard anywhere noticed. The parser is fixed; the guard was never built, so the next edit to either side re-opens it silently.
• ⚠︎ AND IT IS NOT A GUARD ON THE SCORER ITSELF. The kit's whole apparatus checks the ARM against the key; nothing checks the grading code against the key's own stated conventions. Every gate here was green while a scorer bug reported a working model as 0 of 60.
• It is not a claim about real packages. Every figure is agreement with a computed key over an invented corpus of 60 generated packages.
• There is no human-in-the-loop mechanism in the kit. The form is output; a person checking and signing it is the workflow this kit assumes and does not implement.
• There is no retry policy. One call in sixty died in transport and its package's 14 rows are counted wrong. That handling is honest and it is not free.

WatchedWhat is watched, and why that one

The kit measures 43 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

50 measured by the latest run -7 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (answerable_rows 800, calls_attempted 60, calls_in_totals 60, calls_missing_from_totals 0, cite_rows 800, dataset_version licence-application-v1-60packages, derived_rows 240, failures 0, failures_at_ceiling 0, fields 840, fields_answered 840, format_invalid_rows 5, licence_packages 60, mandatory_rows 840, near_miss_rows 195, not_found_rows 40, packages_answered 60, planted_rows 40, resettled_from_cache False, socket_timeout_s 1200, stated_rows 600) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

3 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

extraction · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 50 chips that all say so.

extraction · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 54 chips that all say so.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 50 chips that all say so.

DeviationsWhat deviated

0 breaches across 3 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

The corpus generator, the label gate, the free floor, the stub arm and the local board all run free, with no key and no network, so the whole non-paid half can be re-run on every change. Re-scoring the committed cache is free too, and that is now enforced rather than promised: API_KEY= python3 -m evals.run --run-id r001-licence-application --resume --rescore calls NOTHING BY CONSTRUCTION. ⚠︎ IT DID NOT UNTIL 2026-08-31. This run cached 59 replies for 60 ids, so the sixtieth fell straight through into the call list: with a key configured a re-score would have silently re-bought the package, and without one it aborted before writing any result file, so the promised free repair was unreachable either way. It now carries forward the failure this run id already recorded instead of inventing a new one or dropping the package, which is why the denominator stays at 60. ⚠︎ AND --rescore WITHOUT --resume STILL RE-FIRES ALL SIXTY -- the free path is nested inside the resume branch. The paid arm has run once, on 2026-08-31, and been graded twice.

In one lineNo framework, deliberately

NO FRAMEWORK. A folder of readable Python, standard library end to end, and five JSON files. requirements.txt names nothing third-party and says why. There is no agent, no chain, no orchestration and no state between packages -- one call in, fourteen rows out, and every decision after the reply is code you can read.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Linear, and concurrent only at the package level: read the package, build five prompt parts, one call, parse, normalise, validate, locate every citation, score. There is no branch, no retry loop and no second pass -- a failed call is recorded as a failure and its package's fourteen rows are counted wrong.

The other sideWhat a framework costs you

• Adding a provider means editing the adapter by hand. In exchange requirements.txt stays empty and a forker runs this on whichever key they already hold.
• The reply is parsed by hand. It held across all 59 answered calls; a structured-output library would have failed loudly where this fails quietly.
• No validation library, so the contract between data/schema.json's format_note and src/schema.py::parse is prose on one side and code on the other, with nothing comparing them. This kit is what that costs: they disagreed for one grading and it read as 59 wrong field rows, a 91.3 pct value score and a 0-of-60 form score, with every citation credited and every why correct. Correcting the parser -- 0 calls, $0.00 -- returned the same replies to 826 of 840 values and 57 of 60 forms. The two sides agree today and STILL nothing compares them.
• No retry policy, so a transport failure is a lost package rather than a slower one. One of the sixty calls died that way.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-licence-application on the fast tier, 2026-08-31. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 1 earlier run on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

No drift chart. Only one run has recorded a call latency; the other 1 run on record made no model call at all — a rules-only floor stores 0, which is an absence, not a zero-millisecond answer. One timed run is a reading, not a trend, and it is in the table above. A chart appears the day a second run times itself.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — each document goes whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, FLOOR_RUN, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT for the kit's code and its generated corpus -- the MIT text is in the kits repository's LICENSE-PUBLIC, NOT in its root LICENSE, which reads PROPRIETARY AND CONFIDENTIAL. The kit is MIT; the repository is not. There is no third-party data here to licence and no collected or scraped material of any kind: the corpus, the schema, the control list and the answer key are all generated in process. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe field row -- value, outcome and citation, all three right, one row per schema field

whether each of the 840 field rows is one a filer could sign: the right value on its stored form, the right one of FOUND / DERIVED / NOT-FOUND / FORMAT-INVALID, and a citation that points at the keyed evidence. A field the arm returned no row for is scored wrong rather than dropped -- an application with a field silently absent is the failure this kit exists to catch, not a smaller denominator.

MethodHow the test was run

python3 -m evals.run --run-id <id> [--floor rules]; evals/scoring.py compares against data/gold.jsonl. No model is in the grading path.

Every grader on these pages scored the same 840 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl -- 840 field rows and 1,165 citation spans computed from tools/build_corpus.py's injected facts at SEED = 20260831 and read back off the RENDERED package, every span asserted to slice to the exact words the key quotes. Never typed by a person. Graded independently by evals/check_labels.py, which imports nothing from src/: its own document splitter, its own label reader, its own money parser, its own date arithmetic, its own reading of the control list, and a DIFFERENT rule for the schedule -- the floor says a goods line carries a part number, the checker says it has a whole number in the quantity column and an amount in the extended-value column, so a line continued without a part number is goods to one and not to the other. Re-run free this session: KEY CLEAN over all 840 rows and all 1,165 spans. Every company, party, part number, amount, date, registration number, authorisation reference, end use and control code is invented -- see data/SOURCES.md.

No true/false rates for this grader. It records 12 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No tolerance on codes, enums, counts or money -- a declared value out by one cent is wrong. The single stated tolerance is on free text, case-folded and whitespace-collapsed with a trailing full stop dropped, applied identically to every arm. The citation threshold is recall >= 0.5 AND precision >= 0.5, both published, and one of the model's two non-transport misses sits at recall 0.462 -- so the band is real and the denominator (787 cited rows) supports it.

Cadence: Every run, on every arm. Free, and re-derivable from the committed cache for $0.00 -- which is how this run's model figures were produced: API_KEY= python3 -m evals.run --run-id r001-licence-application --resume --rescore, 0 calls.

The decisionWhen to reach for it

In one lineThe FORM -- every one of a package's 14 fields right, or the form is wrong

whether the whole application would be filed correctly. An application is filed whole: one field wrong is a form that bounces, or worse, one that does not. This is the unit the kit leads with and it is the one that separates the arms here -- the free floor 30 of 60 (50.0 pct), the model 57 of 60 (95.0 pct) on value, outcome AND citation, or 58 of 60 (96.7 pct) on value and outcome alone.

MethodHow the test was run

the same run; it is a second denominator over the same graded rows.

Every grader on these pages scored the same 840 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The same data/gold.jsonl, read whole: a form is correct when all 14 of its rows are. The 60/840 split is fixed by the key before any arm answers.

No true/false rates for this grader. It records 10 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No band at all: 14 of 14 or the form is wrong. The denominator is 60, so one package is 1.7 points and no finer claim than that is supportable. Both arms' by-case breakdowns are published on their own denominators (25 clean, 5 per planted case) rather than as one rate, because 5 is too small to carry one.

Cadence: Every run. It is the same graded rows under a second denominator and costs nothing.

The decisionWhen to reach for it

In one lineThe two error directions -- invented and abandoned, never averaged

which of the two mistakes an arm is making. INVENTED: the key says the package does not answer the field and the arm filled it anyway, from a value belonging to somebody else -- denominator, the 40 NOT-FOUND rows. ABANDONED: the key carries a value and the arm answered NOT-FOUND -- denominator, the 800 answerable rows. They are not worth the same: an abandoned field comes back from the office that receives the form, and an invented one does not come back at all.

MethodHow the test was run

the same run; the two rates are computed on separate denominators and there is no blended figure anywhere in this kit.

Every grader on these pages scored the same 840 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The same data/gold.jsonl. The split into 40 NOT-FOUND rows and 800 answerable ones is fixed by the key before any arm answers, and so are the 195 rows carrying a recorded near-miss and the 40 planted rows.

No true/false rates for this grader. It records 10 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No tolerance and no blended figure anywhere. The two directions have denominators of 40 and 800 and are never averaged. ⚠︎ THE 40 IS SMALL: one invented row is 2.5 points, so no band finer than that is supportable and the count is published beside every rate.

Cadence: Every run, on every arm. It is the lens to read when a headline accuracy figure is quoted.

The decisionWhen to reach for it