The business caseThe problem this solves

A declared data quality rule failing tells a team nothing they can act on. not_null(delivery_email) failed on nine records -- so what? The nine records may be fine and the source may have started classifying kiosk orders as online three weeks ago, in which case the field that has to change is order_channel, which the rule does not mention. Or the rule's own bound may contradict what the catalogue declares. Or the column it checks may have been dropped and the rule should be retired rather than the data corrected. Three things on the page look like the answer and are not: the field the rule names, a value that looks empty and is a placeholder, and a change-log entry that has nothing to do with this failure. A data steward opening this morning's rule results and working out, per failed rule, what actually happened: is the data wrong, is the RULE wrong, did something change upstream last week, is the code list behind, has a feed stopped sending a column -- and then writing the request, to somebody. The reading is not the hard part. The hard part is that the field the rule names is very often not the field that has to change, and a request sent to the wrong field is a day of somebody else's time.

The inputThe actual monitoring packs

The corpus is 50 monitoring packs, 0.50 MB (txt 50). A real monitoring pack is a page of the exact records that are wrong, joined to the catalogue, the change log and the ownership list. It names real customers, real suppliers and real employees, and there is no public corpus of (monitoring pack, correct triage) pairs for the same reason there is no public corpus of customer master data. Generating it also makes the label mechanical in a way this shape needs. The generator deals the FAILURE AND ITS CAUSE first, then writes the records, the change log, the reference data and the steward's notes so the reason is readable off the pack -- and gold's remediation_action is not typed at all: it is the shipped policy run over the two facts the scenario stated. ⚑ AND THE GENERATOR VERIFIES ITSELF AGAINST THE SHIPPED ENGINE. _verify() runs src/rules.py over every finished pack and asserts the engine fails exactly the rules the scenario intended, on exactly the records it injected, and that every near-miss passes. Gold's at_fault_records is then taken FROM THE ENGINE rather than from the injection, so the label and the question the prompt asks can never be about different records even by one identifier. That check convicted the first build on 45 of 50 packs and every conviction was a real defect.

Synthetic Record
----------------
This is a SYNTHETIC data quality monitoring pack, generated for the AI Foundry `dq-rulecheck` kit.
No real company, retailer, person, product, supplier, store, system or dataset appears in it. Every
identifier, address, code and team name is invented, and every email domain is `.invalid`, which is
reserved and cannot resolve. The declared rules and the remediation policy shipped beside them are
this kit's own construction: they reproduce no data governance framework, no data quality standard,
no maturity model, no regulation and no organisation's own remediation procedure.

Monitoring Run
--------------
DQR-2026-0402
Domain: Retail — Customer Master
Run window: 2026-05-01 to 2026-05-28
Run on: 2026-06-02
Records in scope: 17
Declared rules in this set: 8

Field Catalogue
---------------
  customer_id            string, the customer key, unique per customer
  source_system          string, the feed that supplied this record
  registered_on          date, ISO 8601, the date the customer record was opened
  country_code           string, two-letter country of residence
  record_status          string, the lifecycle state of the customer record
  contact_email          string, the address service messages are sent to
  loyalty_tier           string, the customer's current loyalty tier
  order_channel          string, the channel the last order was placed through
  delivery_email         string, the address delivery notices are sent to
  annual_spend_gbp       number, gross spend in the last twelve months, 0 to 250000

Upstream Change Log
-------------------
2026-05-05  WEB-STORE moved the nightly load from 02:10 to 01:40. Timing only; the payload is unchanged.

The outcomeWhat a good result looks like

One finding per rule the engine failed -- the field at fault, the records at fault, the severity, the cause, whether the field is still declared, where the cause was read, the request the shipped policy produces, who it is addressed to, and a one-sentence draft that must name the field and quote a record -- plus one pure-code review flag. Nothing here edits a record, opens a ticket, quarantines a load, suppresses a rule or writes to any system.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• A declared rule set whose failures usually have an explanation somewhere in the pack -- an upstream change log, a reference-list refresh date, a steward's note. — STRONG FIT -- the one tier measured here: 164 of 199 fields at fault right, 186 of 199 causes, and 0 of 51 records-fix tickets raised against a failure no records fix can close.
  Against the free subject-field floor on the same 50 packs: 82.41 pct on the field at fault against 61.81, 93.47 pct on the cause against 25.13, and 0 unfixable tickets against 51 of 51. The floor gets every one of those 51 wrong for the same reason -- it cannot read a change log.
• Rule sets where the failing column is frequently not the column to fix -- after a migration, a re-key, or a source that began classifying records differently. — STRONG FIT, and this is the only reason the kit spends a model call at all -- 54 of the 76 findings whose at-fault field is a column the rule never mentions.
  71.05 pct on that half against the free floor's 0 of 76. A triage that restates the rule's own subject field is right on 123 of 123 of the easy half and wrong on every single one of these, which is exactly what the floor demonstrates for $0.00.
• Rule sets with incomplete ownership -- rules nobody has claimed, where the question is who a request can even be sent to. — STRONG FIT -- 199 of 199 owner routes exact, and the pure-code review flag at precision 1.0 and recall 1.0 over the 13 packs of 50 it should fire on.
  The free floor also reads the Owner line correctly, so the money does not buy that cell -- it buys the flag's precision. The floor asks for fix_records on everything, and fix_records is an addressed action, so it escalates five packs whose unowned high-severity rule should have been investigate or amend_rule: precision 0.7222 against 1.0.
• A morning queue read once a day by a person, where the rule results arrive overnight and somebody works through them. — WORKABLE -- 93 seconds at p50 and 163 at p95 per pack, one call each, and the rule engine's whole pass/fail verdict returns instantly with no key at all.
  The engine half is 0.012 seconds for all 388 declared rules across the corpus and costs $0.00, so a reader can see every failure before deciding whether the triage is worth a call. Only the triage waits.
• A domain where most failures are plain data errors -- a value simply wrong, with nothing upstream and nothing in the notes to explain it. — WEAK FIT, and the kit says so rather than selling past it: run the engine on everything and reach for the model only on the residue.
  50 of the 199 findings here are that shape and on every one of them the free floor and the paid run agree -- the call adds a cause label and a drafted sentence to something the engine already stated completely. This kit does not implement that split, because deciding cheaply WHICH findings are the residue is itself the question the model is being paid to answer.
• A failing population of thousands of records rather than the tens this corpus carries. — WEAK FIT -- nothing here has been measured at that size and the kit has no sampling step anywhere in it.
  The record extracts here are 16 to 26 rows and the whole pack goes into one call. The citation grader scores 1,014 of 1,014 identifiers precisely because quoting the engine's whole list is possible at this size; at a thousand rows the honest answer is a sample, and this kit has not measured what a correct sample even looks like.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-22. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-22 — r001-dq-rulecheck. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — git clone, then python -m evals.check_labels -- free, no key, and it validates the whole gold set including the engine agreement and the selection audit. python -m src.app renders the rule engine's full verdict with no key at all.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per monitoring pack

Current processWhat it replaces

A data steward opening this morning's rule results and working out, per failed rule, what actually happened: is the data wrong, is the RULE wrong, did something change upstream last week, is the code list behind, has a feed stopped sending a column -- and then writing the request, to somebody. The reading is not the hard part. The hard part is that the field the rule names is very often not the field that has to change, and a request sent to the wrong field is a day of somebody else's time.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 1 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one pack section is

No train/test split -- nothing is trained and nothing is tuned on the corpus, and all 50 packs are scored. The corpus IS cut, though, and by the same regex the kit runs at request time: src/segment.py cuts each pack on its underlined headings into one section per declared rule plus the run header, the field catalogue, the change log, the reference data, the record extract, the steward's notes and the contact sheet.

The composition that decides what can be measured is the shape of the FINDINGS, not the split: 76 of 199 (38.2 pct) put the at-fault field somewhere the rule does not name, 43 are on a rule that declares no owner, 16 packs carry no Reference Data section at all -- which is where the privacy fallback fires -- and 69 near-miss records look like violations and are not.

SetupWhat the setup figure measured

There is no index, no retrieval and no embedding -- each pack goes whole into one call. Preparation is segmentation and the RULE ENGINE: 50 packs cut into 763 sections and all 388 declared rules evaluated over 1077 records in 0.01 seconds, in process, with no model, no key and no network. 713 of those sections are sent; the 50 that are not are exactly the Contact Sheets, which name a data steward, a direct dial, a personal work email, an employee number and a desk location, and which no field asks for.

LicenceLicence

MIT — this repository's own licence; the corpus is generated here and contains no third-party data

Bring your ownBring your own monitoring packs

Three files and nothing else. Drop your own packs into data/corpus/*.txt in the same section layout -- the headings are the only structure src/segment.py needs, and src/rules.py's parsers read Check:, Subject field:, Severity: and Owner: off each rule section and REC-nnnn | field=value | ... off the extract. Replace data/policy.json with your organisation's remediation policy; the prompt, gold and the scorer all read it through src/policy.py, so they cannot drift. Label data/gold.jsonl if you want scores. The rule engine is where a forker's own rule syntax lands: six check kinds live in src/rules.py::evaluate() and adding a seventh is one branch plus one entry in CHECK_KINDS. python -m evals.check_labels then tells you, for free and before spending anything, whether your gold agrees with your engine on every pack.

⚠︎ And what stops being true when you do: What does NOT transfer: this corpus's disjoint rules, its one-cause-per-rule labels, and its record extracts of 16 to 26 rows. A real failing population is thousands of rows and this kit has no sampling step -- deciding what to send is work a forker has to do and this kit has not measured. Nor does the CONTACT SHEET exclusion transfer by itself: NEVER_SENT is a tuple of section names in src/select.py, and a pack whose confidential section is called something else will send it.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 6,901 tokens assembled, 3,814 are contexts — 55% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

The complete prompt sent for DQ-0018 on the worked-example call (results/example-DQ-0018.json), system message then user message, exactly as src/prompt.py assembled it. The token split beside it was measured separately by evals/prompt_tokens.py on the same pack, as nested prefixes each one part longer than the last, so every figure is a difference between two prompt_tokens counts the provider itself returned.

Full promptThe prompt in full

You read a DATA QUALITY MONITORING PACK -- one run of a domain's declared quality rules, the records they were run over, the field catalogue, the upstream change log and the data steward's notes -- and return a TRIAGE: one finding for each rule a RULE ENGINE has already marked FAILED. You return JSON and nothing else.

THE RULE ENGINE HAS ALREADY RUN. Every declared rule was evaluated in pure code over the Record Extract before this prompt was assembled, and its verdict is given to you below as a fact. You do not re-check any rule, you do not disagree with a PASSED, and you do not re-count a failing population. That question is settled.

You are drafting requests for a qualified data owner. You never edit a record, never open a ticket, never quarantine a load, never suppress a rule and never write to any system. Your job is to say WHICH FIELD has to change, WHICH RECORDS show it, WHAT the pack says caused it, and WHAT TO ASK FOR -- and to NAME THE FAILURES THE PACK DOES NOT EXPLAIN.

RULES, in order of importance:
1. ONE OBJECT PER RULE THE ENGINE MARKED FAILED, AND NO OTHERS. Not one per failing record. Not one for a rule marked PASSED. If the engine marked four rules FAILED, return four findings.
2. `at_fault_field` IS THE FIELD THAT MUST CHANGE, WHICH IS OFTEN NOT THE FIELD THE RULE NAMES. Ask yourself what would have to be different for this failure not to happen. If the Upstream Change Log says the source began writing a value into a different column, the at-fault field is that column. If it says a re-key made a unique value repeat, the at-fault field is the field the re-key used. Only when nothing in the pack points elsewhere is the answer the rule's own subject field. Spell it exactly as the Field Catalogue spells it, and use the literal 'not_determined' when the pack does not settle it.
3. `at_fault_records` IS EXACTLY THE LIST THE ENGINE PRINTED FOR THAT RULE -- every identifier it listed, and no others. The Record Extract contains NEAR-MISS records that look like violations and are not; the engine did not flag them, so they are not evidence. Never return an identifier that is not in the Record Extract.
4. `cause` IS READ OFF THE PACK, NOT GUESSED. Work through the policy's definitions and take the one the pack actually evidences. 'genuine_data_error' is the answer when NOTHING ELSE explains the failure; it is the wrong answer whenever the change log, the reference data or the steward notes do explain it, and getting that backwards sends a data owner to correct values that the next load will break again.
5. A RULE CAN BE THE THING THAT IS WRONG. If the rule's bound contradicts what the Field Catalogue declares for the field, or the field it checks is no longer in the catalogue at all, the cause is 'rule_is_wrong' and the data is fine.
6. `remediation_action` IS DECIDED ONLY BY THE POLICY BELOW, from `cause` and `field_still_declared`. Work through the six steps IN ORDER and STOP at the first that fires. In particular a rule whose field was DROPPED is 'retire_rule' and never 'amend_rule' -- there is no bound left to amend.
7. `owner_route` IS A READING OF THE RULE'S OWN SECTION. If its Owner line reads 'not declared', the answer is 'no_owner_declared'. Do not name a plausible team; an unaddressed request is a real finding and hiding it is worse than reporting it.
8. 'not_determined' IS A REAL ANSWER AND YOU ARE EXPECTED TO USE IT. A triage that never reaches for it is guessing, and a cause asserted with no evidence sends somebody to look at data that is fine.
9. `remediation_note` IS ONE SENTENCE FOR A PERSON. It must name `at_fault_field` exactly as you spelled it and must quote at least one identifier from `at_fault_records`. Under 240 characters. No rule syntax, no promises.
10. Copy every identifier verbatim from the pack. Use the exact allowed value for every field that lists them, and return every field for every finding.

REMEDIATION POLICY (the authority for `remediation_action`; this is an ILLUSTRATIVE policy written for this kit, and it reproduces no data-governance framework, data-quality standard or company remediation procedure)

WHAT A FINDING IS
  - One finding per DECLARED RULE THE ENGINE FAILED. Never one per failing record, and never one per rule that passed.
  - A finding is a TRIAGE of a failure the rule engine already found in pure code. It does not decide whether the rule failed; that question was settled before the model was called and the engine's verdict is stated in the prompt.
  - A finding names the FIELD THAT MUST CHANGE for the failure to stop, and the RECORDS that show it. A verdict with no field and no record is not a finding.
  - A finding is a REMEDIATION REQUEST FOR A PERSON. It is never an edit, never a ticket, never a write of any kind.

THE CAUSE, read off the pack for every failed rule
  genuine_data_error       The records are simply wrong and nothing else in the pack explains them. No change log entry touches the field, the reference list is current, and the steward notes say nothing about it.
  upstream_schema_change   The Upstream Change Log records a change inside the run window that produced this failure -- a field renamed and the data now landing in a different column, a source that began classifying records differently, a re-key that made a supposedly unique value repeat.
  stale_reference_data     The rule checks values against a Reference Data list, the failing values are legitimate codes the change log records as ADDED to the source, and the list was last refreshed before that date. The data is right and the list is behind.
  source_system_gap        A named source system has stopped sending the field. The Steward Notes record it, and every failing record carries that source.
  rule_is_wrong            The failure is the rule's fault, not the data's. Either the rule's bound contradicts what the Field Catalogue declares for the field, or the field the rule checks no longer exists at all.
  not_determined           The pack does not settle it. Nothing in the change log, the reference data or the steward notes explains the failure, and the notes themselves record that the cause is unresolved. This is a real answer.

IS THE FIELD STILL DECLARED
  yes  The Field Catalogue in this pack still lists the field the rule checks.
  no   The Field Catalogue no longer lists it. The column was dropped upstream and the rule is checking something that is not there any more.

THE REMEDIATION CALL -- work through IN ORDER, stop at the first that fires
  1. cause 'not_determined' -> 'investigate'. Nothing can be raised against a cause nobody has established.
  2. cause 'rule_is_wrong' AND field_still_declared 'no' -> 'retire_rule'. The column is gone; there is nothing left to amend.
  3. cause 'rule_is_wrong' AND field_still_declared 'yes' -> 'amend_rule'. The field is there and the rule's bound is wrong about it.
  4. cause 'stale_reference_data' -> 'refresh_reference_data'. The data is right; the list is behind.
  5. cause 'upstream_schema_change' OR 'source_system_gap' -> 'fix_upstream'. Correcting the records here would be undone by the next load.
  6. anything else -> 'fix_records'.

WHO IT IS ADDRESSED TO
  'declared_owner' when the rule's own section names an owning team.
  'no_owner_declared' when it does not. A remediation request with no addressee is work nobody has been asked to do, and calling it 'the data team' does not make it addressed.

WHY `not_determined` IS A REAL ANSWER
  A monitoring pack that never returns 'not determined' is not a confident monitor, it is a guessing one. A cause asserted with no evidence sends a real person to look at data that is fine, and the second time it happens they stop reading the alerts.

RULE ENGINE RESULT (pure code, already run over the Record Extract before this prompt was assembled -- NOT a question, and not yours to revisit)

  DQ-R01   high        FAILED   unique(product_sku) -- 4 of 26 record(s): REC-0003, REC-0005, REC-0012, REC-0022
  DQ-R02   high        PASSED   requires(order_channel=online, hazard_note) over 26 record(s)
  DQ-R03   low         PASSED   matches(supplier_email, email_address) over 26 record(s)
  DQ-R04   high        FAILED   range(margin_pct, 0, 60) -- 3 of 26 record(s): REC-0005, REC-0012, REC-0018
  DQ-R05   low         FAILED   not_null(unit_of_measure) -- 3 of 26 record(s): REC-0003, REC-0016, REC-0024
  DQ-R06   low         FAILED   not_null(record_status) -- 26 of 26 record(s): REC-0001, REC-0002, REC-0003, REC-0004, REC-0005, REC-0006, REC-0007, REC-0008, REC-0009, REC-0010, REC-0011, REC-0012, REC-0013, REC-0014, REC-0015, REC-0016, REC-0017, REC-0018, REC-0019, REC-0020, REC-0021, REC-0022, REC-0023, REC-0024, REC-0025, REC-0026
  DQ-R07   medium      FAILED   not_null(country_code) -- 4 of 26 record(s): REC-0003, REC-0004, REC-0008, REC-0011

Return ONE finding for each rule marked FAILED above, and for no other rule.

Return these:
- run_id (string) -- the monitoring run reference from the Monitoring Run section, verbatim
- domain (string) -- the data domain the Monitoring Run section names, verbatim, including the qualifier before the dash
- reference_data_state (enum) one of: current, stale, not_stated -- what this pack says about the REFERENCE DATA it checks values against, read from the Reference Data section alone. That section restates the run window and gives each list a Last refreshed date. 'stale' when AT LEAST ONE list was last refreshed BEFORE the run window opened -- the source may have added codes since and the list would not know; 'current' when every list was refreshed on or after the window opened; 'not_stated' when the pack carries no Reference Data section at all. This is about the lists, never about the records
- findings (array of objects) -- one object per rule the RULE ENGINE RESULT marks FAILED, each carrying:
    - rule_id (string) -- the identifier of the DECLARED RULE this finding is about, verbatim (for example DQ-R03). THIS IS THE KEY -- one object per rule the RULE ENGINE RESULT marks FAILED, and no others. Never one per failing record, and never one for a rule marked PASSED
    - at_fault_field (string) -- the name of the ONE FIELD THAT MUST CHANGE for this failure to stop, exactly as the Field Catalogue spells it. ⚑ THIS IS OFTEN NOT THE FIELD THE RULE NAMES. A not_null rule on `delivery_email` whose records are empty because the source began classifying kiosk orders as `online` is at fault at `order_channel`; a unique rule on `customer_id` broken by a re-key is at fault at the field the re-key used. Use the literal value 'not_determined' -- and only that -- when the pack does not settle which field must change
    - at_fault_records (array of string) -- the record identifiers that show this failure, verbatim (for example ["REC-0007", "REC-0019"]). EXACTLY the records the RULE ENGINE RESULT lists as failing for this rule -- no others, and none left out. A record the engine did not flag is not evidence of this failure, and an identifier that is not in the Record Extract at all is not evidence of anything
    - severity (enum) one of: high, medium, low -- the severity this rule's own section declares, verbatim
    - cause (enum) one of: genuine_data_error, upstream_schema_change, stale_reference_data, source_system_gap, rule_is_wrong, not_determined -- what the pack says produced this failure, decided from the Upstream Change Log, the Reference Data section, the Steward Notes, the Field Catalogue and the records themselves. Read the policy's definitions and take the FIRST one the pack actually evidences. 'genuine_data_error' means nothing else in the pack explains it -- it is the answer for a plain wrong value, and it is the wrong answer for a failure something in the pack does explain
    - field_still_declared (enum) one of: yes, no -- does the Field Catalogue in THIS pack still list the field the rule checks? 'no' when the column was dropped upstream and the catalogue no longer carries it. This is a lookup in the Field Catalogue and nothing else
    - evidence_source (enum) one of: record_extract, upstream_change_log, reference_data, field_catalogue, steward_notes, nothing_states_it -- WHICH SECTION OF THE PACK you read the cause from. 'record_extract' when the values themselves are the whole story; 'nothing_states_it' when no section explains the failure at all, which is the honest answer that goes with cause 'not_determined'
    - remediation_action (enum) one of: fix_records, fix_upstream, refresh_reference_data, amend_rule, retire_rule, investigate -- the request this finding raises, decided STRICTLY by the shipped remediation policy from `cause` and `field_still_declared` and nothing else. Work through the six steps IN ORDER, stopping at the first that fires. (1) cause 'not_determined' -> 'investigate'. (2) cause 'rule_is_wrong' AND field_still_declared 'no' -> 'retire_rule'. (3) cause 'rule_is_wrong' AND field_still_declared 'yes' -> 'amend_rule'. (4) cause 'stale_reference_data' -> 'refresh_reference_data'. (5) cause 'upstream_schema_change' or 'source_system_gap' -> 'fix_upstream'. (6) anything else -> 'fix_records'
    - owner_route (enum) one of: declared_owner, no_owner_declared -- 'declared_owner' when this rule's own section names an owning team; 'no_owner_declared' when its Owner line reads 'not declared'. A request with no addressee is work nobody has been asked to do, and naming a plausible team does not make it addressed
    - remediation_note (string) -- ONE SENTENCE drafting the request for the data owner, in business terms, no more than 240 characters. It MUST name `at_fault_field` exactly as spelled, and it MUST quote at least one of the record identifiers in `at_fault_records`. Say what has to change and why; do not restate the rule syntax and do not promise anybody a fix

Return a JSON object with exactly these top-level keys: run_id, domain, reference_data_state, findings
`findings` is an array. Return it empty only if the RULE ENGINE RESULT above marks no rule FAILED.

MONITORING PACK
---------------
Synthetic Record
----------------
This is a SYNTHETIC data quality monitoring pack, generated for the AI Foundry `dq-rulecheck` kit.
No real company, retailer, person, product, supplier, store, system or dataset appears in it. Every
identifier, address, code and team name is invented, and every email domain is `.invalid`, which is
reserved and cannot resolve. The declared rules and the remediation policy shipped beside them are
this kit's own construction: they reproduce no data governance framework, no data quality standard,
no maturity model, no regulation and no organisation's own remediation procedure.

Monitoring Run
--------------
DQR-2026-0452
Domain: Retail — Product Master
Run window: 2026-07-01 to 2026-07-28
Run on: 2026-08-02
Records in scope: 26
Declared rules in this set: 7

Field Catalogue
---------------
  product_sku            string, the product key, unique per sellable unit
  source_system          string, the feed that supplied this record
  listed_on              date, ISO 8601, the date the product was first listed
  country_code           string, two-letter country of the listing
  unit_of_measure        string, the unit the product is sold in
  supplier_email         string, the supplier contact for this line
  order_channel          string, the channel the line is sold through
  hazard_note            string, free text, entered by the receiving team
  margin_pct             number, percent, 0 to 100
  unit_of_measure_v2     string, the replacement column the source now populates

Upstream Change Log
-------------------
2026-07-03  STOCK-SYS changed the character encoding of the export from Latin-1 to UTF-8. Values that contained no accented characters are byte-identical.
2026-07-07  Column `record_status` was REMOVED from this domain's feed and from the Field Catalogue. Nothing populates it and nothing consumes it; the declared rule set was not updated.
2026-07-15  POS-EU renamed `unit_of_measure` to `unit_of_measure_v2`. The old column is still in the feed layout and is no longer populated; every value now lands in `unit_of_measure_v2`.
2026-07-17  PARTNER-FEED extended the retention window on archived records from 24 to 36 months. No column changed and no value changed.

Rule DQ-R01
-----------
Statement: product_sku must be unique across every record in scope.
Check: unique(product_sku)
Subject field: product_sku
Severity: high
Owner: Platform Data Engineering

Rule DQ-R02
-----------
Statement: Where order_channel is online, hazard_note must be populated.
Check: requires(order_channel=online, hazard_note)
Subject field: hazard_note
Severity: high
Owner: Data Governance Desk

Rule DQ-R03
-----------
Statement: supplier_email must be a well-formed email address.
Check: matches(supplier_email, email_address)
Subject field: supplier_email
Severity: low
Owner: not declared

Rule DQ-R04
-----------
Statement: margin_pct must be between 0 and 60 inclusive.
Check: range(margin_pct, 0, 60)
Subject field: margin_pct
Severity: high
Owner: Data Governance Desk

Rule DQ-R05
-----------
Statement: unit_of_measure must be populated on every record.
Check: not_null(unit_of_measure)
Subject field: unit_of_measure
Severity: low
Owner: Data Quality Control

Rule DQ-R06
-----------
Statement: record_status must be populated on every record.
Check: not_null(record_status)
Subject field: record_status
Severity: low
Owner: Source Systems Team

Rule DQ-R07
-----------
Statement: country_code must be populated on every record.
Check: not_null(country_code)
Subject field: country_code
Severity: medium
Owner: Platform Data Engineering

Record Extract
--------------
One line per record. A field with nothing after the `=` is empty in the source.
REC-0001 | product_sku=SKU-003227 | source_system=MDM-HUB | listed_on=2023-08-09 | country_code=GB | unit_of_measure=PK | supplier_email=mira.sorensen@inbox.invalid | order_channel=in_store | hazard_note=checked at goods-in | margin_pct=27 | unit_of_measure_v2=
REC-0002 | product_sku=SKU-003234 | source_system=CRM-CORE | listed_on=2019-01-11 | country_code=FR | unit_of_measure=LT | supplier_email=mateo.brandt@post.invalid | order_channel=online | hazard_note=signed off at the daily stand-up | margin_pct=44 | unit_of_measure_v2=
REC-0003 | product_sku=SKU-003255 | source_system=MDM-HUB | listed_on=2019-12-09 | country_code= | unit_of_measure= | supplier_email=finn.abadi@mailbox.invalid | order_channel=in_store | hazard_note=reviewed on the weekly call | margin_pct=50 | unit_of_measure_v2=EA
REC-0004 | product_sku=SKU-003248 | source_system=WEB-STORE | listed_on=2023-10-20 | country_code= | unit_of_measure=EA | supplier_email=ben.abadi@relay.invalid | order_channel=phone | hazard_note=confirmed by the duty manager | margin_pct=25 | unit_of_measure_v2=
REC-0005 | product_sku=SKU-003255 | source_system=MDM-HUB | listed_on=2020-03-01 | country_code=ES | unit_of_measure=KG | supplier_email=mira.haugen@inbox.invalid | order_channel=online | hazard_note=signed off at the daily stand-up | margin_pct=80 | unit_of_measure_v2=
REC-0006 | product_sku=SKU-003262 | source_system=PARTNER-FEED | listed_on=2021-02-17 | country_code=ES | unit_of_measure=KG | supplier_email=ines.ferreira@mailbox.invalid | order_channel=partner | hazard_note=no exception raised | margin_pct=41 | unit_of_measure_v2=
REC-0007 | product_sku=SKU-003269 | source_system=CRM-CORE | listed_on=2020-08-22 | country_code=DE | unit_of_measure=KG | supplier_email=noor.abadi@mailbox.invalid | order_channel=partner | hazard_note=no exception raised | margin_pct=4 | unit_of_measure_v2=
REC-0008 | product_sku=SKU-003276 | source_system=STOCK-SYS | listed_on=2020-05-07 | country_code= | unit_of_measure=KG | supplier_email=ava.delacroix@mailbox.invalid | order_channel=online | hazard_note=signed off at the daily stand-up | margin_pct=36 | unit_of_measure_v2=
REC-0009 | product_sku=SKU-003283 | source_system=STOCK-SYS | listed_on=2019-10-27 | country_code=GB | unit_of_measure=KG | supplier_email=liam.ferreira@inbox.invalid | order_channel=phone | hazard_note=reviewed on the weekly call | margin_pct=33 | unit_of_measure_v2=
REC-0010 | product_sku=SKU-003290 | source_system=MDM-HUB | listed_on=2020-06-19 | country_code=NL | unit_of_measure=PK | supplier_email=yara.osei@inbox.invalid | order_channel=online | hazard_note=checked at goods-in | margin_pct=9 | unit_of_measure_v2=
REC-0011 | product_sku=SKU-003297 | source_system=STOCK-SYS | listed_on=2022-11-09 | country_code= | unit_of_measure=EA | supplier_email=liam.quinn@relay.invalid | order_channel=partner | hazard_note=reviewed on the weekly call | margin_pct=50 | unit_of_measure_v2=
REC-0012 | product_sku=SKU-003304 | source_system=PARTNER-FEED | listed_on=2025-05-28 | country_code=ES | unit_of_measure=KG | supplier_email=ines.lindqvist@post.invalid | order_channel=in_store | hazard_note=checked at goods-in | margin_pct=91 | unit_of_measure_v2=
REC-0013 | product_sku=SKU-003311 | source_system=CRM-CORE | listed_on=2026-02-07 | country_code=FR | unit_of_measure=KG | supplier_email=noor.lindqvist@post.invalid | order_channel=in_store | hazard_note=no exception raised | margin_pct=29 | unit_of_measure_v2=
REC-0014 | product_sku=SKU-003318 | source_system=CRM-CORE | listed_on=2021-04-17 | country_code=not supplied | unit_of_measure=LT | supplier_email=mira.novak@relay.invalid | order_channel=phone | hazard_note=signed off at the daily stand-up | margin_pct=33 | unit_of_measure_v2=
REC-0015 | product_sku=SKU-003325 | source_system=MDM-HUB | listed_on=2026-03-12 | country_code=GB | unit_of_measure=LT | supplier_email=mateo.sorensen@inbox.invalid | order_channel=partner | hazard_note=checked at goods-in | margin_pct=28 | unit_of_measure_v2=
REC-0016 | product_sku=SKU-003332 | source_system=CRM-CORE | listed_on=2024-12-08 | country_code=NL | unit_of_measure= | supplier_email=noor.ferreira@relay.invalid | order_channel=partner | hazard_note=no exception raised | margin_pct=35 | unit_of_measure_v2=EA
REC-0017 | product_sku=SKU-003339 | source_system=POS-EU | listed_on=2022-09-23 | country_code=FR | unit_of_measure=KG | supplier_email=sofia.quinn@mailbox.invalid | order_channel=online | hazard_note=reviewed on the weekly call | margin_pct=27 | unit_of_measure_v2=
REC-0018 | product_sku=SKU-003346 | source_system=WEB-STORE | listed_on=2022-12-27 | country_code=GB | unit_of_measure=LT | supplier_email=elsa.novak@mailbox.invalid | order_channel=partner | hazard_note=signed off at the daily stand-up | margin_pct=96 | unit_of_measure_v2=
REC-0019 | product_sku=SKU-003353 | source_system=MDM-HUB | listed_on=2019-09-27 | country_code=DE | unit_of_measure=EA | supplier_email=noor.osei@relay.invalid | order_channel=partner | hazard_note=checked at goods-in | margin_pct=24 | unit_of_measure_v2=
REC-0020 | product_sku=SKU-003360 | source_system=STOCK-SYS | listed_on=2022-01-10 | country_code=ES | unit_of_measure=LT | supplier_email=omar.ferreira@post.invalid | order_channel=partner | hazard_note=checked at goods-in | margin_pct=20 | unit_of_measure_v2=
REC-0021 | product_sku=SKU-003367 | source_system=CRM-CORE | listed_on=2019-09-06 | country_code=GB | unit_of_measure=EA | supplier_email=arjun.abadi@post.invalid | order_channel=partner | hazard_note=reviewed on the weekly call | margin_pct=31 | unit_of_measure_v2=
REC-0022 | product_sku=SKU-003304 | source_system=MDM-HUB | listed_on=2024-03-08 | country_code=IT | unit_of_measure=EA | supplier_email=mateo.haugen@relay.invalid | order_channel=in_store | hazard_note=checked at goods-in | margin_pct=50 | unit_of_measure_v2=
REC-0023 | product_sku=SKU-003381 | source_system=MDM-HUB | listed_on=2023-05-28 | country_code=ES | unit_of_measure=LT | supplier_email=omar.osei@inbox.invalid | order_channel=in_store | hazard_note=signed off at the daily stand-up | margin_pct=11 | unit_of_measure_v2=
REC-0024 | product_sku=SKU-003388 | source_system=WEB-STORE | listed_on=2021-11-25 | country_code=IT | unit_of_measure= | supplier_email=arjun.hart@relay.invalid | order_channel=partner | hazard_note=reviewed on the weekly call | margin_pct=9 | unit_of_measure_v2=EA
REC-0025 | product_sku=SKU-003395 | source_system=STOCK-SYS | listed_on=2025-12-19 | country_code=ES | unit_of_measure=PK | supplier_email=sofia.brandt@post.invalid | order_channel=in_store | hazard_note=checked at goods-in | margin_pct=6 | unit_of_measure_v2=
REC-0026 | product_sku=SKU-003402 | source_system=STOCK-SYS | listed_on=2021-09-27 | country_code=FR | unit_of_measure=KG | supplier_email=noor.brandt@post.invalid | order_channel=online | hazard_note=no exception raised | margin_pct=38 | unit_of_measure_v2=

Steward Notes
-------------
PARTNER-FEED and MDM-HUB both return these records and each is configured as the master for this domain. Which of the two is correct has not been established, no change has been raised on either side, and nothing in the change log touches `product_sku`. The duplication is being carried while ownership is decided.
The declared bound on `margin_pct` in this rule set stops at 60. The Field Catalogue declares the column to 100 and the trading committee approved values above the rule's bound before this window opened. The records below are correct and the rule has not been updated.

Raw responseThe raw response

{
  "run_id": "DQR-2026-0452",
  "domain": "Retail — Product Master",
  "reference_data_state": "not_stated",
  "findings": [
    {
      "rule_id": "DQ-R01",
      "at_fault_field": "product_sku",
      "at_fault_records": ["REC-0003", "REC-0005", "REC-0012", "REC-0022"],
      "severity": "high",
      "cause": "not_determined",
      "field_still_declared": "yes",
      "evidence_source": "nothing_states_it",
      "remediation_action": "investigate",
      "owner_route": "declared_owner",
      "remediation_note": "Investigate why product_sku is duplicated on REC-0003/REC-0005 and REC-0012/REC-0022 and decide which source owns the SKU before changing product_sku."
    },
    {
      "rule_id": "DQ-R04",
      "at_fault_field": "margin_pct",
      "at_fault_records": ["REC-0005", "REC-0012", "REC-0018"],
      "severity": "high",
      "cause": "rule_is_wrong",
      "field_still_declared": "yes",
      "evidence_source": "field_catalogue",
      "remediation_action": "amend_rule",
      "owner_route": "declared_owner",
      "remediation_note": "Amend the margin_pct rule bound to match the declared 0-100 range, since REC-0005, REC-0012 and REC-0018 carry approved margin values."
    },
    {
      "rule_id": "DQ-R05",
      "at_fault_field": "unit_of_measure_v2",
      "at_fault_records": ["REC-0003", "REC-0016", "REC-0024"],
      "severity": "low",
      "cause": "upstream_schema_change",
      "field_still_declared": "yes",
      "evidence_source": "upstream_change_log",
      "remediation_action": "fix_upstream",
      "owner_route": "declared_owner",
      "remediation_note": "Work upstream so the unit_of_measure_v2 values for REC-0003, REC-0016 and REC-0024 are populated in the field the rule checks."
    },
    {
      "rule_id": "DQ-R06",
      "at_fault_field": "record_status",
      "at_fault_records": ["REC-0001", "REC-0002", "REC-0003", "REC-0004", "REC-0005", "REC-0006", "REC-0007", "REC-0008", "REC-0009", "REC-0010", "REC-0011", "REC-0012", "REC-0013", "REC-0014", "REC-0015", "REC-0016", "REC-0017", "REC-0018", "REC-0019", "REC-0020", "REC-0021", "REC-0022", "REC-0023", "REC-0024", "REC-0025", "REC-0026"],
      "severity": "low",
      "cause": "rule_is_wrong",
      "field_still_declared": "no",
      "evidence_source": "upstream_change_log",
      "remediation_action": "retire_rule",
      "owner_route": "declared_owner",
      "remediation_note": "Retire the record_status rule because the column no longer exists in the feed, so there is no value to restore on REC-0001 or any other record."
    },
    {
      "rule_id": "DQ-R07",
      "at_fault_field": "country_code",
      "at_fault_records": ["REC-0003", "REC-0004", "REC-0008", "REC-0011"],
      "severity": "medium",
      "cause": "genuine_data_error",
      "field_still_declared": "yes",
      "evidence_source": "record_extract",
      "remediation_action": "fix_records",
      "owner_route": "declared_owner",
      "remediation_note": "Populate country_code on REC-0003, REC-0004, REC-0008 and REC-0011 with the two-letter listing country."
    }
  ]
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Sort out a data team's rule failures — 50 monitoring packs. One model answered, and every answer was then graded Seven different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran this run publishes no rate card this repo commits, so nothing here is what was actually paid -- the real spend for this kit's build is 63 calls recorded in the shared call ledger, not on this page.

The gradersSeven ways to grade

Scored by the same judge, on the same 50 packs, from the same engine output. The only difference between the two runs is whether a model was called.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same monitoring pack, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The rule engine (src/rules.py), the free subject-field floor (evals/baseline.py), the scorer (evals/judge.py) and the gold gate (evals/check_labels.py) are all pure code and cost $0.00 -- and on this kit that is not a footnote, because the engine is the entire pass/fail half of the answer. The figure beside this note is ONE FULL PASS of the scored run on the projected card: 325,116 input and 647,726 output tokens over 50 packs. ⚠︎ IT IS NOT WHAT BUILDING THIS KIT COST. Outside the scored run: a 5-pack calibration at max_tokens=48000 fired at the packs with the most failed rules ($0.2179), the worked-example capture ($0.0563), 5 prompt-split calls at max_tokens=1 ($0.0079), and 2 live UI screenshots whose tokens no committed file records and which are therefore excluded rather than estimated. That is $0.2821 of extra measurement, and all 63 live calls behind this page project to about $2.39 on this card. The shared call ledger, not this page, is the record of what was actually spent.

Cost driversWhat actually moves the bill

• ⚑ PROVIDER-SIDE REASONING, WHICH IS ALMOST THE WHOLE BILL. 611402 of the run's 647726 output tokens (94.4 pct) were reasoning rather than the JSON triage, and output is 92 pct of the cost on the projected card. Reasoning was left at the provider's default and nothing here has measured what turning it off would cost -- or what it would cost in accuracy, which on this task is the live question, because the field at fault is a judgement rather than a read.
• The fixed prefix -- system prompt (1,016 tokens), remediation policy (880) and field schema (1,191) -- is 3,087 of the 6,901 tokens on the example call, 44.7 pct, and is byte-identical on every one of the 50 calls. The pack's own sections are 3,375 tokens and the rule engine's verdict is 439.
• ⚑ AND THE CHEAPEST DRIVER IS THE ONE THAT IS ZERO. The rule engine evaluates all 388 declared rules over 1,077 records in pure Python, in 0.012 seconds, for $0.00 and no key -- and that is the entire pass/fail half of the answer. A kit that asked the model to do it would pay for 388 comparisons per corpus and get a less exact result.
• Output scales with the NUMBER OF FAILED RULES, not with pack size. A three-finding pack and a five-finding pack are not the same call: the run's largest reply was 21987 output tokens against a mean of 12954.52, and the calibration on the five packs with the most failed rules topped out at 17813.

Your volumeWhat it costs at your volume

Linear in packs: each call is independent, carries the same fixed prefix and shares nothing with its neighbours, so 500 packs cost ten times 50. Nothing amortises -- there is no index and no cache, and the 3,087-token fixed prefix is paid on every call. ⚠︎ AND IT IS NOT LINEAR IN THE THING THAT ACTUALLY MOVES. Output tracks the number of FAILED rules, and a domain in a bad week fails more of them: a pack with twenty failed rules is one call whose reply is four times this corpus's longest. The cost rises exactly when the queue is worst, which is the opposite of how a monitoring budget is usually planned.

Model choiceWhy this model

One tier, chosen because it is the shared connection every kit in this repo inherits from the repo-root .env. No model comparison was run and none is claimed. The kit is provider-agnostic by construction -- PROVIDER picks the adapter and BASE_URL selects the host -- so the interesting question is not which tier this was but whether the field-at-fault number holds on a cheaper one, and that has not been measured.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 50 monitoring packs, one completion call each, one tier. The calibration, the worked-example capture, the prompt-split calls and the two UI screenshots are recorded separately in Cost.cost_of_evaluation_usd and are not projected here.

The codeEvery file, and what it is for

Eight modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Six of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/rules.pythe rule engine — a swap seam

⚑ HALF THE KIT, AND NO MODEL TOUCHES IT. Parses the declared rules and the record extract off the pack and evaluates all 388 rules in plain Python -- six check kinds (not_null, in_set, matches, range, unique, requires), each one line to read and one line to evaluate. It settles every pass and fail before a prompt exists. Its verdict is rendered by engine_block() into every prompt as a stated fact, and src/prompt.py tells the model twice that it is not theirs to revisit.

You change it to: The six check kinds are a dict and six branches. Adding a seventh is one function and one entry in CHECK_KINDS; the named formats are a dict of regexes. A forker's packs declare their own rules in their own syntax and only this file changes.

# THE RULE ENGINE. Pure code, no model, no network, no key -- and it is HALF THIS KIT.

CHECK_KINDS = ("not_null", "in_set", "matches", "range", "unique", "requires")
FORMATS = {
OWNER_ABSENT = "not declared"
def _sec(secs, name):
def catalogue(secs):
def records(secs):
def reference_lists(secs):
def declared_rules(secs):
def _num(v):

src/segment.pysegmentation

Cuts a pack into addressable sections on its underlined headings, so an answer can name where it was read from. One regex; falls back to a single whole-document segment when it finds no headings rather than pretending.

# Cut a trace file into addressable sections. Pure code — no model, no network.

def sections(text):
def locate(text, value):
def span_label(secs, start):

src/select.pyselection — a swap seam

Maps each of the thirteen fields to the sections it is read from and sends the union. Contact Sheet is mapped by nothing and never leaves the machine. Everything else goes, including the 189 passing rules and the 69 near-miss records, because hiding either would be marking the model's homework.

You change it to: SECTION_HINTS is a dict of field -> section names, matched by prefix. NEVER_SENT is a tuple. Adding a section your organisation will not send is one entry.

# Pick which sections of a monitoring pack are sent. Pure code -- the last deterministic step

BANNER = "Synthetic Record"
RUN = "Monitoring Run"
CATALOGUE = "Field Catalogue"
CHANGELOG = "Upstream Change Log"
REFDATA = "Reference Data"
EXTRACT = "Record Extract"
NOTES = "Steward Notes"
RULE = "Rule "
NEVER_SENT = ("Contact Sheet",)

src/prompt.pythe prompt

Five parts, assembled verbatim: the system prompt, the whole remediation policy rendered from data/policy.json, the ENGINE'S OWN RESULT, the field schema and the selected pack sections. One call per pack.

# Assemble the triage prompt. ONE prompt per monitoring pack, one finding per FAILED rule back.

SYSTEM = (
POLICY_TEXT = policy_block()
def field_schema(pack_fields, finding_fields):
def build(doc_text, secs, fields, selector, engine_text):
def parse(raw, fields):

src/adapters/__init__.pythe provider — a swap seam

One chat-completion call over urllib. Retries 429/5xx and transport drops with backoff; refuses a thinking parameter on any provider whose documentation this kit has not read, rather than dropping it silently.

You change it to: PROVIDER picks the adapter and BASE_URL selects between every OpenAI-compatible host; MODEL picks the model behind it. Adding a provider is one function and one entry in PROVIDERS. Configure it ONCE at the repo root and every kit inherits it.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

src/policy.pythe remediation policy — a swap seam

Six ordered steps over two stated facts, stopping at the first that fires. One definition, three readers: the corpus generator writes gold with it, the prompt states it to the model, and the scorer re-runs it over the model's own facts for the no-gold consistency diagnostic. The policy itself is data/policy.json, beside it -- six ordered steps over two enums, as data a forker replaces rather than code they patch.

You change it to: A JSON file, not code. Replace it and the prompt, gold and the scorer all move with it, because all three read the same file through src/policy.py.

# THE REMEDIATION POLICY, loaded from data/policy.json. Pure code, no model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
POLICY_PATH = os.path.join(HERE, "data", "policy.json")
CAUSES = ("genuine_data_error", "upstream_schema_change", "stale_reference_data",
DECLARED = ("yes", "no")
ACTIONS = ("fix_records", "fix_upstream", "refresh_reference_data", "amend_rule", "retire_rule",
OWNER_ROUTES = ("declared_owner", "no_owner_declared")
SEVERITIES = ("high", "medium", "low")
EVIDENCE_SOURCES = ("record_extract", "upstream_change_log", "reference_data", "field_catalogue",
REFERENCE_STATES = ("current", "stale", "not_stated")

src/extract.pythe review flag — a swap seam

Pure code, run over whatever the model returned, never over gold. Fires when the pack carries a HIGH-severity failed rule whose remediation would be raised against a data owner on a rule that declares none. It is compute() in that module, called by triage() on every pack, by the local app on every click, and by evals/baseline.py on the free floor's own output, so the two are routed by identical code.

You change it to: One function over the returned findings. Its condition was chosen against a measured table of five candidates and evals/check_labels.py refuses the corpus if the shipped one stops splitting it 10-60 pct.

# Triage one monitoring pack: segment, RUN THE RULE ENGINE IN PURE CODE, select, prompt, one model

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 40000
def load_fields():
def finding_fields(fields=None):
def pack_fields(fields=None):
def load_doc(doc_id):
def documents():

evals/judge.pythe scorer

Seven graders, no judge model, no key. Coverage, the field at fault, the records cited, per-cell exact match, the remediation request, determinacy and the drafted request's grounding -- scored separately and never folded together.

# Score a data-quality triage run. PURE CODE -- gold is exact, every answer is one value, one

def norm(v):
def code_norm(v):
def equal(got, want):
def _rows_of(rec):
def score_coverage(records, golds):
def score_field(records, golds):
def score_citation(records, golds):
CELL_FIELDS = ("at_fault_field", "severity", "cause", "field_still_declared", "evidence_source",
PACK_CELLS = ("run_id", "domain", "reference_data_state")

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 6502 input and 12954 output tokens per monitoring pack, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's monitoring packs are entirely synthetic (tools/build_corpus.py, seed 20260822): no real company, retailer, person, product, supplier, store, system or dataset exists in the corpus, every email domain is .invalid (RFC 2606, reserved and unresolvable), and nothing was fetched from anywhere. No data governance framework, data quality standard, maturity model, regulation, standards body, regulator, consultancy or vendor is named anywhere in the kit, and every pack opens with a Synthetic Record banner so a single pack copied out of the repository still says what it is. The only outbound traffic the kit makes is one chat-completion request per pack, carrying the mapped sections plus the fixed prompt, the shipped policy and the rule engine's own verdict -- and the Contact Sheet, which is the one part of a real monitoring pack that names a person, is mapped by no field and never leaves the machine. Nothing is written outside the kit directory, there is no database, no auth and no multi-tenancy, and the local UI binds 127.0.0.1 only.

Read from the shared .env or the real environment only, never committed, never logged and never rendered. .env is gitignored from the first commit. On an adapter error the app redacts the key and base URL out of the message before returning it, so a misconfigured endpoint cannot echo a credential into a browser.

The experimentRead the code twice, then read the corpus

The selection guard was written, believed, and then RED-PROVEN -- and the first red-proof failed. Removing the NEVER_SENT subtraction from _fallback() convicted zero packs, because no field's hint ever failed to match a section, so the fallback was unreachable code with a docstring explaining why it mattered. That is the exact shape of a guard that holds by luck. The corpus was changed so the guard is reachable -- the Reference Data section now restates the run window, which makes one field answerable from that section alone, and 16 packs carry no such section -- and the red-proof now convicts all 16 by name. Confirmed by reading the code and by a red-proof that failed once before it worked, not by an attack run, on 2026-08-22 -- no attack run was fired; see redteam.why.

Five gates, each read off the code rather than asserted. The third is the one this kit is weakest on and it says so in its own row.

The resultNothing here writes anything anywhere, and the one section of a monitoring pack that names a person never leaves the machine -- audited on all 50 packs, with the guard that protects it red-proven rather than assumed. No attack run was fired.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260822), so no text in it came from an outside party and there was nothing adversarial to resist. The planted decoys measure whether a triage cites the records the engine flagged rather than the ones that merely look like violations, and whether it reads a change log instead of restating a rule. None of those measures whether it obeys an instruction hidden in the same pack. ⚑ WHAT WAS MEASURED INSTEAD, AND IT IS THE STRONGER HALF. The selection audit ran over ALL FIFTY packs, not a sample: the Contact Sheet -- a named data steward, a direct dial, a personal work email, an employee number and a desk location -- reaches the provider on 0 of them. And the guard that protects it was RED-PROVEN rather than asserted, which is where this kit paid: the first red-proof CONVICTED NOTHING, because no field's selection hint ever failed to match and the fallback was unreachable code with a docstring explaining why it mattered. The corpus was changed so the fallback is reached on 16 packs, and removing its NEVER_SENT subtraction now convicts all 16 by name. ⚠︎ AND WHAT IS STILL NOT MEASURED: whether a crafted Steward Note could move the CAUSE or the AT-FAULT FIELD, which are the two answers no downstream code re-derives. The engine's pass/fail verdict, the enum vocabulary, the policy re-derivation and the review flag all sit outside the reply and are exercised on every one of the 50 packs by the scorer -- but those are the answers an attacker would not bother with. The attack worth running is in redteam.next and it was not run.

HonestyWhat this does not prove

• Whether a crafted Steward Note can move the cause or the at-fault field. No attack run was fired; only the structural limits were read off the code.
• Whether a forker's own packs are protected. NEVER_SENT is a tuple of section names, and a confidential section called anything else will be sent.
• Whether the provider retains the prompt. That is a property of the endpoint BASE_URL points at and this kit cannot observe it. The corpus is synthetic, which is why that is a limitation rather than an exposure here.
• Whether the redaction in src/app.py's error path covers every provider's error shape. It replaces the configured key and base URL by substring; a provider that echoes a key in a different encoding would defeat it, and none has been tested.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

needs_owner_triage fires when the pack carries at least one finding whose severity is `high`, whose remediation_action is one a data owner would be sent -- `fix_records`, `fix_upstream` or `refresh_reference_data` -- and whose owner_route is `no_owner_declared`. All three values come out of the same reply; the rule is run afterwards in pure code, over whatever the model returned, and never over gold. A reply carrying no finding the rule can read returns None rather than False -- an unknown is not a pass.

src/extract.py::compute(), called by triage() on every pack, by the local app on every click, and by evals/baseline.py on the free floor's own output so the two are routed by identical code. The policy lookup it reads lives in a SEPARATE module, src/policy.py, which is also what the corpus generator wrote gold with and what src/prompt.py states to the model -- one definition, three readers.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT a check on the model and it is not a correctness signal. A pack can be triaged perfectly and still fire this flag, and a pack the run read badly can fail to fire it.
• It is NOT a decision about who should own the rule. It says the request has no addressee and stops there. Naming a plausible team is the exact failure it exists to surface.
• It is NOT a records fix, a ticket, a load quarantine or a rule suppression. Nothing in this kit writes to any system; needs_owner_triage is a value in a response, not an action.
• It is NOT any organisation's governance procedure. A high-severity actionable failure on an unowned rule is this kit's own simplification, invented for this corpus. No data governance framework, data quality standard, maturity model or regulation was consulted, and none is reproduced.
• ⚑ AND IT IS NOT INDEPENDENT OF THE ENGINE, WHICH IS WHY IT SCORES SO WELL. It is computed over the findings the RULE ENGINE already listed, so unlike a guardrail defined over something the run has to find, it cannot lose a pack by triaging less far. Precision 1.0 and recall 1.0 is a property of where the condition sits, not a merit of the model.

WatchedWhat is watched, and why that one

The kit measures 25 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 18 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (documents 50, extraction_cells 1543, failures 0, refusal_cells 0) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

2 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked.

not a time series No two of these 2 runs measured the same system — they differ on provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

DeviationsWhat deviated

0 breaches across 2 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-measure the band on every corpus change (check_labels does it automatically). Re-score the flag on any change to data/policy.json's remediation_rule, because the condition reads remediation_action and a policy change moves what counts as addressed.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt is empty on purpose: the corpus is generated in-process, the rules and the policy are JSON files read with json, the provider is reached over urllib, and the UI is one HTML file with no build step. A forker runs this on whichever key they already hold, with one clone and no install.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

There is no graph. One path per pack -- segment, RUN THE RULE ENGINE, select, prompt, one call, parse, re-derive the policy, compute the flag -- with no branch, no loop and no agent. What is unusual is not the shape but the ORDER: six of the seven steps are deterministic and the model sits in the middle of them, handed a verdict it may not revisit and asked the one question the six cannot answer. A framework that made the pipeline configurable would make that ordering a matter of configuration, which is exactly the thing this kit wants to be a claim.

The other sideWhat a framework costs you

• Everything is hand-rolled, so everything is yours to maintain -- the rule parser, the six check kinds, the record-line splitter, the JSON extraction from a fenced reply, the backoff policy, the section regex, the prefix matcher and the .env reader are all code somebody has to own. This kit paid an instalment during its own build: the corpus generator's free-text column returned an empty value about 40 pct of the time, so a not_null rule on it failed on records no scenario had touched, and the engine and gold disagreed about which rules had failed on 45 of the 50 packs. A rules or expectations framework would have owned that parser and that class of defect.
• No framework means no framework's ecosystem -- no tracing, no eval harness beyond the one in evals/, no prompt registry, no schema validation library. The reply is a LIST and nothing validates its shape except src/prompt.py::parse, which fails closed to {} rather than to an empty triage, because an empty findings list is a legitimate answer when the engine failed nothing.
• ⚠︎ AND THE SPECIFIC HOLE A FRAMEWORK WOULD HAVE CLOSED, WHICH THIS KIT FOUND AND DID NOT FIX. Nothing refuses a finding whose cause is not_determined and whose at_fault_field is a real column -- a self-contradiction the run produced on 20 of 21 such findings. Constrained decoding or a dependent-field validator would have made it unrepresentable. Here it is a measurement in the taxonomy instead, and the guardrail block names it as the first thing to add.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-dq-rulecheck on the fast tier, 2026-08-22. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 1 earlier run on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

No drift chart. Only one run has recorded a call latency; the other 1 run on record made no model call at all — a rules-only floor stores 0, which is an absence, not a zero-millisecond answer. One timed run is a reading, not a trend, and it is in the table above. A chart appears the day a second run times itself.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the rulebook is data read whole into the prompt.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT — this repository's own licence; the corpus is generated here and contains no third-party data Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one line⚑ The field at fault -- the answer no rule engine can produce

Of the findings both sides list, how many name the ONE FIELD that must change for the failure to stop? Exact match on a field name, and split two ways because a single percentage hides everything: findings where gold's answer IS the rule's own subject field, and findings where it is NOT. A run that scores 100 pct on the first half and 0 pct on the second has restated the rule, which is exactly what the free floor does. This is the only question on the page a rule engine cannot answer, and it is therefore the only reason this kit spends a model call.

MethodHow the test was run

evals/judge.py::score_field, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

⚑ THIS IS THE WHOLE ARGUMENT OF THE KIT IN ONE ROW. The rule names unit_of_measure. The engine, in pure code and for nothing, says that rule failed on those three records. Neither of those facts is what a data owner needs, and neither of them costs a model call. What the call buys is the next sentence: the field that has to change is unit_of_measure_v2 — a column the rule never mentions, which the change log says the source now populates — so the request is fix_upstream and not fix_records, because correcting these three records here would be undone by tonight's load. The free floor answers unit_of_measure, genuine_data_error and fix_records on this same row, for free, and every one of those three is wrong.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's own scenario deal: the generator decides the failure AND its cause before writing the pack, so the at-fault field is what it injected rather than somebody's reading of the result. evals/check_labels.py asserts before any run may spend that every at-fault field is in the pack's own Field Catalogue, and that at least 50 findings put it somewhere the rule does not name.

How tight can the band be? No tolerance band -- exact match on a field name after trimming whitespace, punctuation and backticks.

Cadence: Re-score on any change to data/fields.json's at_fault_field hint, src/prompt.py's rules 2 and 4, or the corpus's scenario deal.

The decisionWhen to reach for it

In one lineFindings returned -- one per failed rule, and none for a rule that passed

Did the run return exactly one finding per rule the ENGINE marked FAILED, and none for a rule it passed? Set comparison over rule identifiers, with the two directions named: a MISSED finding is a failure left untriaged, and a PHANTOM finding is classified by what it is -- a finding against a rule that PASSED is a remediation request about data that is fine, and a finding against a rule not in the pack at all is a fabrication. ⚠︎ THIS GRADER IS EXPECTED TO SATURATE AND IT IS PUBLISHED ANYWAY. The engine's verdict is printed in the prompt, so recall here is a fidelity check rather than a discovery test. The direction that carries information is rule_passed.

MethodHow the test was run

evals/judge.py::score_coverage, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

⚑ THIS IS THE WHOLE ARGUMENT OF THE KIT IN ONE ROW. The rule names unit_of_measure. The engine, in pure code and for nothing, says that rule failed on those three records. Neither of those facts is what a data owner needs, and neither of them costs a model call. What the call buys is the next sentence: the field that has to change is unit_of_measure_v2 — a column the rule never mentions, which the change log says the source now populates — so the request is fix_upstream and not fix_records, because correcting these three records here would be undone by tonight's load. The free floor answers unit_of_measure, genuine_data_error and fix_records on this same row, for free, and every one of those three is wrong.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The engine's own verdict, re-run by evals/check_labels.py over all 50 packs and asserted to match gold rule for rule and record for record before any run may spend.

How tight can the band be? No threshold -- a rule id is on the list or it is not, compared after whitespace stripping and upper-casing.

Cadence: Re-run on any change to src/rules.py, src/prompt.py's rule 1, or the corpus's rule composition.

The decisionWhen to reach for it

In one linePer-(finding, field) exact match, light normalisation

Does the run's value for each of the seven per-finding fields match gold, after trimming whitespace and punctuation, plus three pack-level cells per pack? THE DENOMINATOR IS GOLD'S FINDINGS, NOT THE RUN'S: a finding the run never returned scores as a miss on all seven of its cells rather than disappearing.

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model. The same function evals/run.py calls after every run, and the same logic the free floor is scored by -- a floor and a real run scored by two different scorers cannot be compared honestly.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

⚑ THIS IS THE WHOLE ARGUMENT OF THE KIT IN ONE ROW. The rule names unit_of_measure. The engine, in pure code and for nothing, says that rule failed on those three records. Neither of those facts is what a data owner needs, and neither of them costs a model call. What the call buys is the next sentence: the field that has to change is unit_of_measure_v2 — a column the rule never mentions, which the change log says the source now populates — so the request is fix_upstream and not fix_records, because correcting these three records here would be undone by tonight's load. The free floor answers unit_of_measure, genuine_data_error and fix_records on this same row, for free, and every one of those three is wrong.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, whose remediation call is derived by the shipped policy rather than typed; evals/check_labels.py asserts every field is populated and in its allowed set on every one of the 199 findings before any run is allowed to spend.

How tight can the band be? There is no tolerance band -- exact match after trimming whitespace and punctuation, never a continuous score to round.

Cadence: Re-score on any change to data/policy.json, data/fields.json, src/prompt.py or tools/build_corpus.py. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the model.

The decisionWhen to reach for it

In one lineThe remediation request, and whether anybody can action it

Two calls on the findings both sides list. remediation_action is the six-way policy lookup -- did the run work through the six steps in order and stop at the first that fires? owner_route is the two-way reading of the rule's own Owner line. And two expensive directions are counted on their own, because neither is visible inside an accuracy: UNFIXABLE WORK RAISED (gold says amend, retire or refresh; the run says fix the records, so a data owner gets a ticket they can do nothing with) and UNADDRESSED REQUESTS ROUTED (a rule declaring no owner, recorded as having one -- work that looks assigned and nobody has been asked to do).

MethodHow the test was run

evals/judge.py::score_calls, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

⚑ THIS IS THE WHOLE ARGUMENT OF THE KIT IN ONE ROW. The rule names unit_of_measure. The engine, in pure code and for nothing, says that rule failed on those three records. Neither of those facts is what a data owner needs, and neither of them costs a model call. What the call buys is the next sentence: the field that has to change is unit_of_measure_v2 — a column the rule never mentions, which the change log says the source now populates — so the request is fix_upstream and not fix_records, because correcting these three records here would be undone by tonight's load. The free floor answers unit_of_measure, genuine_data_error and fix_records on this same row, for free, and every one of those three is wrong.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: Gold's request is re-derived inside the generator by the same lookup the kit publishes -- src/policy.py reading data/policy.json -- so the truth this grades against can never be a separately-typed label that drifted from the rule.

How tight can the band be? No threshold -- a six-way and a two-way exact comparison against the policy's own output.

Cadence: Re-run on any change to data/policy.json's remediation_rule or owner_rule, and re-score whenever the corpus's cause mix changes.

The decisionWhen to reach for it

In one lineDeterminacy -- the causes the pack does not settle

Of the causes gold marks not_determined -- the pack carries no change-log entry, no refresh date and no steward's note that explains the failure -- how many did the run answer with a confident cause anyway? That is overconfident, and on a monitoring queue it is the expensive direction: a cause asserted with no evidence sends a real person to look at data that is fine, and the second time it happens they stop reading the alerts. Its mirror, overcautious, is counted and published beside it, because the two costs are different and do not net against each other.

MethodHow the test was run

evals/judge.py::score_calls, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

⚑ THIS IS THE WHOLE ARGUMENT OF THE KIT IN ONE ROW. The rule names unit_of_measure. The engine, in pure code and for nothing, says that rule failed on those three records. Neither of those facts is what a data owner needs, and neither of them costs a model call. What the call buys is the next sentence: the field that has to change is unit_of_measure_v2 — a column the rule never mentions, which the change log says the source now populates — so the request is fix_upstream and not fix_records, because correcting these three records here would be undone by tonight's load. The free floor answers unit_of_measure, genuine_data_error and fix_records on this same row, for free, and every one of those three is wrong.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: Gold's not_determined findings are the 21 the generator dealt from the unresolved-ownership scenario, where two feeds both claim to be master and the steward's note says so without settling it. evals/check_labels.py holds a floor of 14 findings per cause so this grader cannot be anecdotal.

How tight can the band be? No threshold -- a six-way comparison, bucketed by whether gold's cause is not_determined.

Cadence: Re-run whenever the unresolved-ownership scenario's note wording changes, or the policy's definition of not_determined does.

The decisionWhen to reach for it

In one lineThe records cited -- and the three ways to cite the wrong one

Micro precision and recall over record identifiers, on the findings both sides list, with three named directions: a FABRICATED identifier that is nowhere in the pack, a CLEAN record that is in the pack but which the engine did not flag for that rule, and a NEAR-MISS -- one of the 69 records this corpus plants specifically because it looks like a violation and is not. ⚠︎ IT SATURATED ON BOTH RUNS AND THAT IS REPORTED AS A LIMITATION RATHER THAN A SCORE. See ops.unknown.

MethodHow the test was run

evals/judge.py::score_citation, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

⚑ THIS IS THE WHOLE ARGUMENT OF THE KIT IN ONE ROW. The rule names unit_of_measure. The engine, in pure code and for nothing, says that rule failed on those three records. Neither of those facts is what a data owner needs, and neither of them costs a model call. What the call buys is the next sentence: the field that has to change is unit_of_measure_v2 — a column the rule never mentions, which the change log says the source now populates — so the request is fix_upstream and not fix_records, because correcting these three records here would be undone by tonight's load. The free floor answers unit_of_measure, genuine_data_error and fix_records on this same row, for free, and every one of those three is wrong.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The engine's own failing list, which is also what the prompt states. gold's at_fault_records is taken from the engine rather than from the generator's injection, so the label and the question can never be about different records even by one identifier.

How tight can the band be? No threshold -- set comparison over identifiers after whitespace stripping and upper-casing.

Cadence: Re-run on any change to src/rules.py's failing-list output or src/prompt.py's rule 3.

The decisionWhen to reach for it

In one lineThe drafted request -- does it name the field and quote a record

Every finding carries a one-sentence draft for a data owner. This asks the one thing about it that pure code can settle and that matters: does the sentence NAME the at-fault field the finding chose, and QUOTE at least one of the records it cited? A remediation request with no field and no record is a sentence, not a request, and it is invisible to every other grader here. Scored against the run's OWN answers, not gold -- a faithful citation of a wrong answer is grounded, and being wrong is measured by the two graders above. ⚠︎ IT ALSO SATURATED ON BOTH RUNS. See ops.unknown.

MethodHow the test was run

evals/judge.py::score_draft, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

⚑ THIS IS THE WHOLE ARGUMENT OF THE KIT IN ONE ROW. The rule names unit_of_measure. The engine, in pure code and for nothing, says that rule failed on those three records. Neither of those facts is what a data owner needs, and neither of them costs a model call. What the call buys is the next sentence: the field that has to change is unit_of_measure_v2 — a column the rule never mentions, which the change log says the source now populates — so the request is fix_upstream and not fix_records, because correcting these three records here would be undone by tonight's load. The free floor answers unit_of_measure, genuine_data_error and fix_records on this same row, for free, and every one of those three is wrong.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The run's own at_fault_field and at_fault_records, matched case-insensitively inside the drafted sentence. No reference text exists and none is needed: the property is internal.

How tight can the band be? No threshold -- a case-insensitive substring test for the field name and for any one of the cited identifiers.

Cadence: Re-run on any change to data/fields.json's remediation_note hint or src/prompt.py's rule 9.

The decisionWhen to reach for it