All posts

AI document processing9 min read

What Accuracy Checks Should You Require From AI Document Processing?

Before you accept an AI document processing system, require it to be tested on a hand-labelled set of your own documents, with accuracy reported for each field rather than as one headline figure. In live use, every extracted value should carry a calibrated confidence score, link back to where it appears on the page and pass business-rule checks. Values with low confidence or failed checks should go to a person, a random sample of the rest should be audited, and accuracy should be monitored for as long as the system runs.

Why a single accuracy percentage tells you very little

Vendors often quote one number, such as "98% accurate". That number hides most of what matters. An invoice has a supplier name, a date, line items, a VAT figure and a total. A system can read supplier names almost perfectly and still get one total in twenty wrong, and in most businesses the total is the field that does the damage.

Google's Document AI documentation leaves out a plain accuracy metric altogether. Its reasoning is that accuracy means little when some fields are optional and a single label can hold more than one value. Instead it reports three measures:

  • Precision: of the values the system extracted, the share that were correct.
  • Recall: of the values actually on the documents, the share the system found correctly.
  • F1 score: the harmonic mean of the two, combining them into one figure.

The two measures pull against each other. If the system outputs only values it is sure of, precision rises and recall falls. You therefore need both figures for every field that matters, along with a clear rule for what counts as a match. Google's evaluation tool has a fuzzy matching option that ignores case, spacing and punctuation, and removes currency symbols on monetary fields. It does not apply to numbers at all. In Google's own example, 1 and 1.00 do not count as a match. Ask any supplier how matches were scored, and insist that numeric fields were scored on exact values.

Check 1: a held-out test set built from your own documents

Accuracy on a vendor's sample documents does not tell you how the system will handle yours. Require an evaluation on a set of your real documents that a person has labelled by hand and that played no part in building or tuning the system.

The set should cover the variety you actually receive. Microsoft's Document Intelligence guidance says to include every variation of a document, such as scanned and digital PDFs, with at least five samples of each type when a model has to handle both. It also warns that documents which look similar to a person can look different to a model. A representative test set includes:

  • Your highest-volume suppliers, customers or form types.
  • Poor scans, phone photos, rotated pages and faxes, if you receive them.
  • Multi-page documents and tables that run across page breaks.
  • Documents with missing optional fields, so you can see whether the system leaves them blank or invents a value.
  • Older and newer versions of the same template.

Ask for results broken down by document type as well as by field. A system that works on most of your volume but fails on one supplier's layout is a different proposition from one that performs evenly everywhere.

Check 2: confidence scores that mean something

A confidence score is the system's estimate that a value is correct. Microsoft describes field confidence as a probability between 0 and 1, where 0.95 means the value should be right about 19 times out of 20. Confidence is what lets a system decide which values to accept automatically and which to send for review, so it carries much of the weight in any accuracy design. Ask three questions about it.

Is confidence reported per field, and at more than one level?

Microsoft's documentation separates several kinds of confidence: how closely the document matches the expected type, whether the field was found in the right place, and whether each word was read correctly. It advises combining the field and text-reading scores into a composite, because a field can be in the right location and still be misread. Tables have confidence at table, row and cell level. One cell can score highly while its row scores low because another cell in that row was missed.

Where does the confidence come from?

This matters most for systems built on large language models. A Microsoft engineering post on its Content Understanding service calls LLMs "notoriously overconfident". It describes producing confidence with a separate model trained to tell correct extractions from incorrect ones, rather than asking the LLM how sure it is. If a supplier's confidence score is just the model reporting on itself, treat it as weak evidence.

Has it been calibrated against your data?

The same post recommends setting thresholds per field and per document type, based on explicit business objectives, and testing them on held-out documents. In its example, a single threshold caught most errors on some fields while letting many through on others. Ask to see, for each important field, how many errors fall above the chosen threshold. Those are the mistakes that will pass without review.

Check 3: grounding back to the source

Every extracted value should point to the page and position it came from. This is called grounding, and it does two jobs. A reviewer can check a flagged value at a glance without searching the original. And it creates an auditable trail from the record in your system back to the source page.

Grounding also guards against a known failure of generative models: a value that sounds right but is not on the page. As the Microsoft post notes, a model that has misread a value can state it as fluently as a correct one. If the system cannot show where a value appears, it should not be accepted automatically.

Check 4: business-rule validation

Confidence scores measure how sure the model is, not whether the result makes sense. Rules written for your business catch errors the model cannot see. Ask for checks such as:

  • Line items add up to the subtotal, and subtotal plus tax equals the total.
  • Tax rates match the rates that apply in your jurisdiction.
  • Dates are valid and in a plausible range.
  • Supplier names, VAT numbers, IBANs or purchase order numbers match records you already hold.
  • Duplicate documents are detected before they are posted.
  • Required fields are present. A blank total is a failure, not an empty value.

Any value that fails a rule should go to a person, however high its confidence.

Check 5: human review with clear triggers

Human review is the safety net, and it works best when the triggers are written down. Amazon's documentation for using Textract with its Augmented AI review service lists four triggers: a specific field falls below a confidence threshold, a specific field is missing, any field falls within a set confidence range, or a document is picked at random for sampling. Amazon states that Augmented AI is no longer open to new customers, but the same trigger logic applies to any system.

The random sample matters more than it looks. Routing on low confidence only catches the errors the system knows it might be making. Sampling documents that were accepted automatically is how you find confident mistakes. Require:

  • Documented triggers for each field, agreed with you rather than set by the supplier alone.
  • A review screen showing the extracted value beside the highlighted source.
  • A record of every correction, kept as data for measuring and improving the system.
  • A stated sampling rate for auto-accepted documents, which can be lowered once results justify it.

Check 6: independent cross-checking for high-stakes fields

Where one wrong value is costly, you can require a second, independent opinion before anything is accepted. In an AI bookkeeping pipeline we built for Pixelet, several AI models process each entry independently, and an entry is accepted only when they agree. The design started from the view that in bookkeeping, a confident wrong answer is worse than no answer, so no single model gets the final say.

More generally, disagreement between independent models is a useful signal, because it marks the items most worth a person's attention. Running several models means more processing per document, so reserve cross-checking for the fields where the cost of an error justifies it.

Check 7: monitoring after go-live

Accuracy at launch tells you little about accuracy six months later. Suppliers change their invoice layouts, new form versions appear and scan quality varies. Microsoft notes that reported accuracy can be inconsistent when live documents differ from training documents, and that low document-type confidence points to template or structural variations. The Content Understanding post recommends continuous monitoring and recalibrating thresholds as new form versions and document sources appear. Ask for:

  • A regular report of field-level error rates, based on corrections and random samples.
  • Alerts when the share of documents sent for review, or the average confidence for a field, moves beyond an agreed range.
  • A process for adding new layouts to the test set and re-running the full evaluation after any change to models, prompts or thresholds.
  • Version records, so any extracted value can be traced to the model and configuration that produced it.

What to put in the acceptance criteria

When you commission or buy a system, turn these checks into written acceptance criteria:

  1. A held-out, hand-labelled test set from your own documents, covering every document type and quality level you receive.
  2. Precision and recall reported per field and per document type, with exact matching for numbers.
  3. Minimum targets for each critical field, set by what an error would cost you.
  4. Field-level confidence from a calibrated method, with thresholds tested on held-out data.
  5. A source location for every extracted value.
  6. Business-rule validation, with failures sent for review.
  7. Written human review triggers, including random sampling of auto-accepted documents.
  8. Monitoring, alerts and re-evaluation after every change.

Ownership decides how far you can enforce these over time. If the thresholds, rules and test set sit inside a vendor's platform, you can only change what the vendor allows. Khive builds AI document processing systems in code and hands them over, so the client owns the system, including its rules and routing logic, and can tighten them as their documents change.

Send us a brief

If you plan to automate document handling and want accuracy checks built in from the start, send us a brief. Tell us which documents you process, roughly how many, which fields matter most and what a wrong value costs you, and we will explain how we would test and check them.

Sources

Related: AI document processing at Khive

Contact

Book a call

Pick a 30 minute slot and we will talk through what your team does by hand and what an agent could take on. Or send a brief: we reply within 24 hours with how an agent would handle it and a straight read on scope.

What Accuracy Checks Should You Require From AI Document Processing? | Khive