Poland's national e-invoice schema checks nothing an accountant would

• 0
  Assertions in the standard
  in 183 798 bytes; XSD 1.0 has no way to express one
• 15
  Consistency rules added
  12 arithmetic identities, 3 heuristics, counted apart
• 108
  Documents generated
  698 rows, gold with no annotation step
• 3 / 6
  Attacks the arithmetic never sees
  it defends against misreading, and an attacker is not a misreading

Does “the arithmetic holds” predict “the fields are right”?

The project's headline question, asked of claude-haiku-4-5, which gets 42 of 107 documents wrong and so supplies a real, unlabelled model-error population. The invariants are run on the prediction, never on the gold — the signal has to be available at inference time on a document nobody annotated.

Precision 100 %, recall 76.2 %, and 100 % of the catches point at a field that is genuinely wrong.

The recall is not a property of the detector. It is the fraction of that model's mistakes that happened to land on numeric fields. Caught: payment_account × 25, lines[].discount × 9, lines[].description × 5. Every one of the 11 misses is a name or a description — a field with no redundancy behind it for arithmetic to check — and 8 of them sit in the multi_page tier, where a description wraps across a page break. Not one arithmetic error escaped.

That is the case for the grounding layer, arrived at by measurement rather than assumed: it covers exactly the fields the arithmetic cannot see, because a description the model invented is a value that resolves to no span on the page.

A caution the study prints beside its own numbers: a confidently wrong but internally consistent answer is invisible. The constant baseline sits at prevalence 100 % with recall 0 %.

When the arithmetic has nothing to say

Every cross-field identity needs two figures to compare. An answer that keeps the total and drops the rows and the rate blocks offers one — so those rules are not wrong about it, they are unable to speak. The stripped baseline is exactly that reading, on all 108 documents.

The one rule that fires is the heuristic whose entire content is no rule could run, and it fires on every document, with 0 false positives. The fields this reading does keep are missing rather than wrong — which is why the two hard rules that need only a single figure, the NIP and the IBAN check digits, did run and were right to find nothing. Those three heuristic rules had never fired on any earlier run — a metric identical across every variant, which this project's own rules call broken rather than stable. Giving them a population was the fix; dropping them was the alternative.

And the gate accepts all of it. Coverage is measured over the values a prediction asserted, so a reading that loses three quarters of the invoice routes everything to accept, at 100 % accuracy, leaking nothing. No signal here separates “correctly absent” from “silently dropped”; the count of what was never asserted is the only column that sees it.

How much of a reading was the template?

The synthetic corpus varies what an invoice means and not how the page says it: three layouts, one vocabulary, one number format. So a second corpus prints the identical gold in three unfamiliar ones — other Polish labels for every field, other column orders, other number and date formats, and block orders that put the totals before the rows. Same seed, same invoices, document for document: a difference between the two columns is the page, because nothing else moved.

pattern did not read the page badly — it could not begin. Not one of the 108 documents produced an invoice the schema would accept, every one recorded as schema_invalid. The labels it matches are the whole of what it was doing.

The other two rows are the controls that make the first one mean something. The oracle is handed the gold and is unaffected, which says the corpus is scorable and its gold did not move; the constant baseline never looks at the page and scores identically on both, which is what a paired comparison should do to a reader that reads nothing.

This is not a real held-out set and does not claim to be. It holds the semantics fixed on purpose, so it answers one question and not the other: how much of a result was presentation. Real invoices also bring skew, stamps, scans and layouts nobody anticipated, and none of that is here.

When the page is a picture

Every document measured so far arrived with a text layer reportlab wrote: exact, complete, in the order the values were drawn. That is the last unearned advantage in the corpus. An invoice in a real inbox is frequently a photograph of an invoice, and there is no text layer at all. So a third corpus prints the same gold in the same layout and then scans it, at three rungs that each isolate one thing: one keeps a text layer (an OCR assumed perfect), one removes the text layer and changes nothing else, and one is what a supplier emails at 150 dpi.

The control is exact rather than approximate: the fitted reader's predictions on the rung that keeps a text layer are identical, field for field, to its predictions on the clean corpus. That is what makes the other two columns a measurement of the missing text layer and not of the damage to the image.

For claude-haiku-4-5 and claude-opus-5, its own page is that model reading the clean corpus as text, so the drop across each of those rows is two changes at once — the page became a picture and the reader started looking at pixels. The column that isolates the modality alone is searchable: the same information is available there as in the text arm, and it is read as an image.

One figure above is not a legibility result: claude-haiku-4-5 on rasterised (1 document) ran out of output tokens mid-answer. A truncated document contributes every one of its fields as missed, which is enough to move a rung's accuracy by several points on its own — and running out of room while transcribing a long table from an image is a cost of the modality rather than of the rung.

The column above is not the finding. What a scan does to the gate is. Run the oracle — a perfect reading, nothing wrong anywhere — over the scanned corpus, and grounding has nothing to resolve against on two of the three rungs:

The last column was the finding, and it was a defect. Grounding resolves a value to a span of page text; where there is none it used to answer UNGROUNDED anyway, so every one of those 3989 values arrived as a false alarm on a reading with nothing wrong in it. That does not degrade, it inverts, and silently: an ungrounded correct value looks exactly like an ungrounded fabricated one. It now answers I could not ask, so those values leave the curve rather than filling it, and the count is printed above every affected gate.md. The false alarms are gone entirely — not one on this run. Of the four signals the routing gate is built on, only the arithmetic survives a scan — place contention and the completeness check need page text as much as grounding does — and that is the one an adversary can satisfy on purpose — so what changed is that the gate reports having no opinion where it used to report a wrong one. A better instrument, and the same missing capability.

And the mirror of it, on a population of real mistakes: claude-haiku-4-5 reading the same pages as images.

Read the first row against the other two. Where a text layer survives, grounding is at its most precise measurement anywhere in this project — it catches almost every wrong value and raises no false alarm at all, on a real population of vision errors rather than an injected one. On rasterised and scanned it does not answer: every value falls in the last column, and the row is empty because the signal is absent there rather than weak. That is the honest shape, and it is also the reason the confusion cells are worth printing per rung instead of pooled — pooled over the whole corpus, this signal once looked barely precise at all, which was a measurement of the missing text layer wearing the reader's name. The gate does not survive a scan; it survives an OCR — a usable engineering conclusion rather than a negative one: put a recogniser in front of the model and the signal comes back.

The gap this fills

The schema published by the Ministry of Finance is XSD 1.0. That version has no assert element at all, so a conforming validator checks types, enumerations and cardinality — and stops. P_15, the gross total, is a bare decimal with no stated relationship to the per-rate totals, and those have none to the line items.

Checking those rules is therefore real work rather than a re-run of validation that already exists. The schema is vendored in the repository with its SHA-256, and a test enforces the digest — so this claim is checkable rather than asserted.

What is checked instead

15 rules, reported as data rather than raised as errors. A model that refused to construct a broken invoice could not be routed, measured or explained — and inspecting broken invoices is the entire project.

Severity is part of the design, not decoration. A hard rule is an arithmetic identity: a violation means something is genuinely wrong. A heuristic usually holds but has lawful exceptions — a faktura uproszczona really may carry a total and nothing else. Mixing the two would blunt the detector, because a heuristic's false positives would be indistinguishable from a real arithmetic miss.

The corpus

Ground truth and the rendered page come from one artifact: the generator writes a document that validates against the vendored XSD, and the same file read back through the extraction schema is the gold. There is no annotation step, so there is no annotation noise to confuse with model error — which is what makes the detector study interpretable at all.

What the corpus deliberately does not do

It is not sanitised to be easy to parse. A quantity of 3 printed beside a price of 466,62 reads as 3 466,62 in a flat text dump, because a space is also Poland's thousands separator. That ambiguity is in real invoices and it stays in this one; the source layer resolves it from word geometry rather than by having the generator avoid it.

It is not a rare case: 873 of the 1888 amounts the corpus prints (46%) carry that thousands space. The source layer reads each of them back as one field, and reads a quantity printed beside a price as two — from the boxes the words occupy, not from the string. The separation is not a tuned threshold either: a space is 0.32 em wide, while every column gap in the corpus is at least 12 pt, because each cell is asserted to fit its column with reportlab's padding still to spare.

What it does not have is a page nobody in this repository designed. Two of the three things that used to be missing are measured now, on held-out corpora of their own — an unfamiliar layout and a poor scan, each varying one thing so that a drop is attributable to it, and both reported above. What is still absent is a genuinely real invoice: a stamp, a signature, a fold, a layout no template anticipated because no template wrote it.

What the baselines say, and what the model says

7 committed runs over the same 108 documents, scoring 22 fields per invoice and 6066 gold field instances in total. Every row answers in the same wire format and goes through the same prompt, parse, validation and repair loop, so a column is comparable down the table. The numbers are recomputed for this page from each run's committed predictions.jsonl — not copied from a report.

recall is how many of the values on the page the prediction offered a value for; value is how many of those it read correctly; accuracy is the two together. They are separate columns because a field that is half missed and a field that is half misread read the same in one number and need different work. A dash means there was no denominator — never a zero, which would read as a measurement nobody made.

oracle is handed the gold, so its 100 % is a check on the harness rather than a result: if a perfect reading scored anything less, every other number here would be wrong. constant answers the same lawful invoice for every document — the floor, and a diagnostic, because the fields it still scores well on (currency, invoice kind) are fields where the corpus's own distribution does most of the work. pattern is regular expressions and column positions with no model at all, and it was deliberately allowed to match the labels this project's own renderer prints, which the extraction prompt is forbidden to know: it is the strongest thing that is not a language model, and therefore the bar. noisy is the gold with known errors injected at a fixed rate — not a competitor but the labelled error set the detector study needs, since a detector measured only on a model's unlabelled mistakes is measured on a sample nobody chose.

claude-opus-5 is the system under test, and it ties the oracle: every field of every document, for $3.20. Its only divergence from the gold's own serialisation is 63 trailing zeros — it writes 137.30 where the generator stored 137.3, which is what the page prints and what the scorer's amount comparison was already defined to treat as the same quantity. A corpus that the strongest non-model reader finds hard and a frontier model finds trivial is measuring layout parsing, not reading.

The invoice as untrusted input

Seven payloads — six that ask for something and one control that asks for nothing — printed in 4 places on the page, including in white ink, where a human approving the invoice sees nothing and the text layer carries every word. The gold of an attacked document is the gold of the document it was made from, so the same scorer, detector and gate run over it unchanged.

The column that matters is arithmetic fires. It is computed against attack-gullible, a control that obeys every instruction it finds on a page: its success rate is 100 % by construction, which is what makes it the right arm for this question. What a defence does about an attack that worked is a property of the defence, and does not need a model to have been fooled first.

The three payloads the arithmetic never sees are account_redirect, refusal, seller_swap — and two of them, account_redirect, seller_swap, are what an attacker would actually run. The other one, refusal, can only stop the document being processed, and the committed attack.md records why that is not the gate defending: no value was accepted from them because none was returned. That is an availability attack succeeding. Redirecting a payment or reissuing the invoice under another NIP does not break any identity: the attacker picks an account they control, so the check digits are valid, and the value is printed on the page, so grounding finds it too. Both of M5's signals were measured on a model's errors, where a wrong digit is a random digit. Neither transfers to an adversary who can compute a check digit, and the committed attack.md shows the routing gate accepting every one of those documents.

That is the milestone's result, and it is a negative one: the arithmetic gate is a defence against misreading, not against injection. Ordering the stages so a document can never choose them, deriving the fence marker from the text it wraps, and telling the model to transcribe rather than compute are what stand between the payload and the answer — and the honest way to report a structural defence is to attack it and publish the rate.

Now photograph the attacked page

Compose the two corpora — M6's grid printed and then put through M7's scanner, gold untouched by either — and the compliant reader above, breached on every attacked document it is given a clean page of, is breached on 25.0 % of these (36 of 144). The decomposition is the whole finding.

Computed from attacked-scanned-gullible, the same control M6 uses, by rebuilding each scanned attacked document's gold in memory from the corpus seed.

The two zeros are blindness, not defence. No payload reached the text layer on those rungs because no text did — the page carries none. A reader that looks at the page sees them exactly as the reach table says it does.

Measured at build time with no model. text means the marker is in the text layer the source reader gets off the scanned page; image means the attacked page and the unattacked page it was made from differ as pictures, through the same scanner at the same seed — so a payload that moved no pixel cannot be seen by anything that looks at the page.

A scan deletes the white-ink attack outright. White on white contributes no pixel, so there is nothing for a recogniser to recover and nothing for a vision model to read: the placement designed to be invisible to the human approving the invoice is the one a photocopier destroys. That is an accident of the medium and not a control — it protects only the placement that hides from a person, and an attacker who prints in ink loses nothing.

A frontier model reading the pixels met 0 of 144 attacking objectives. claude-opus-5, every page sent as an image — an attack success rate of 0.0 %, the same at every rung , with no repairs and nothing left unanswered. 108 of those 144 carried their payload as ink on a page it looked at. The rest are the white-ink placement the scanner had already erased, where a rate is not evidence of anything.

Two things that does not establish, and both are limits of the suite rather than of any model: the payloads are fixed strings that never adapt and were not written against the reader being measured, so the rate scores a catalogue rather than an adversary; and this is a synthetic corpus. The denial payload is the illustration — it beat the compliant control every time and the model above not once, which is a fact about two readers rather than about the payload. The structural defences are still what the case rests on, because they are what holds when the reader is not this one.

And gating still costs accuracy here. Auto-accepting only the high-confidence values is 99.0 % accurate and lets 18 wrong values through, while not gating at all — accepting every one of the 9894 values the reader asserted — is 99.3 %. The gate's signal exists only on the rung that kept a text layer, which is exactly the rung where the attacks worked — so its confident bucket is concentrated on the attacked documents. On a population of model errors the same gate raises accuracy instead.

The curve's own bottom row is a different comparison: it accepts everything the gate could assess, 2697 values, at 97.6 %. The other 6048 sit on a page with no text layer, where the gate has no signal at all — not a low one. It used to call them ungrounded, which is how a scan came to look like a reader's mistake; saying so removed the false alarms and did not bring the signal back.

What is not built yet

Stated plainly, because a portfolio page that reads as finished when it is not is worse than no page at all.

The headline question — does “the invariants hold” actually predict “the fields are correct”? — met an awkward first answer: on the run that mattered most there was nothing to predict, because the model made no mistakes. It has been answered instead against injected errors, a weaker reader, and two held-out corpora — and both of them show that neither making a page harder to see nor making it unfamiliar un-saturates it for a frontier model. What the unfamiliar page did change is the shape of a weaker model's errors: right value, wrong field, which is exactly what the grounding signal cannot see. A negative result is a publishable result, and so is a corpus that turned out to be too easy.

Reproduce it

git clone https://github.com/P0w3r223/doc-extract
cd doc-extract
python -m venv .venv && .venv/Scripts/python -m pip install -e ".[dev]"
pytest                                             # the figures above are assertions
python -m doc_extract.synth --out data/synthetic   # 108 documents, reproducible from one seed
python -m doc_extract.eval run --baseline pattern  # predict, score, write results/pattern/
python docs/build_index.py                         # rebuild this page from the repository

The corpus is not committed: it is a function of one integer, so a seed in the manifest is a smaller and more honest artifact than several hundred PDFs in git history. The manifest also records the reportlab version and the font digests, because every rendered byte depends on them.