Accuracy Benchmarks

Results sourced from committed evaluation artifacts. Dates shown are the measurement dates.

OCR Engine Upgrade: PP-OCRv5 → PP-OCRv6

Measured 2026-07-09 on a local RTX 3090 GPU running the production image (paddleocr==3.7.0, CUDA 12.9). Subjects: a real 24 MP HEIC camera document (Polish text, human-transcribed ground truth) and the repository sample invoice/receipt PDFs (pdftotext ground truth). Scoring is order-insensitive multiset recall and precision.

The deployed configuration is PP-OCRv6 medium with text_det_box_thresh=0.5. The lower box threshold recovers faint lines from gloss/perspective regions in camera photos (+20 pt word recall) with zero regression on clean rendered pages.

Metric	PP-OCRv5 (previous)	PP-OCRv6 medium + box_thresh 0.5 (current)
Camera-photo word recall	0.670	0.865 (+29 %)
Camera-photo Polish-diacritic recall	0.562	0.775 (+38 %)
Camera-photo word precision	0.986	0.955
Camera-photo numeric recall	0.000	0.000
Receipt numeric recall	0.818	1.000 (+22 %)
Dense printed invoice (word recall / precision)	1.000 / 1.000	1.000 / 1.000
Warm GPU latency (RTX 3090, dense page)	~0.55 s	~1.13 s

Word recall on printed rendered pages (invoice and receipt PDFs) is 1.000 on both engine versions — the improvement is concentrated on camera photos where perspective, gloss, and compression stress the detector. Numeric recall (amounts, dates, identifiers) is the closest proxy for field-extraction accuracy, since fields are matched from those tokens.

Photo Engine: Camera-Document Preprocessing

Measured 2026-07-09 (same hardware and image as the OCR engine bench). The photo engine applies geometric unwarping and orientation classification to camera uploads before OCR. The decision was to enable it for IMAGE uploads only — unwarp applied to already-flat rendered pages causes a regression from 1.000 to 0.635 word recall by distorting aligned pixels.

Metric	Without preprocessing (baseline)	With unwarping (camera uploads)
Camera-photo word recall	0.865	0.921 (+6.6 %)
Camera-photo Polish-diacritic recall	0.775	0.854 (+10.2 %)
Camera-photo word precision	0.955	0.961
Dense printed page word recall	1.000	0.635 (regression, photo-only by design)

The unwarping gain is real but requires the OCR service to return the transformed page image so bounding boxes are drawn in the correct (transformed) coordinate space. The photo-only restriction prevents the dense-page regression. The feature is gated by the OCR_PHOTO_TRANSFORMS_ENABLED environment variable.

Multi-Invoice PDF Split Detection

Measured 2026-07-21 using the synthetic labeled evaluation set generated by scripts/gen_split_eval.py and scored by scripts/eval_doc_split.py. The detector uses the embedded text layer (pdftotext) to find page-boundary signals (invoice headers, date-label patterns, document-type keywords) before any OCR or LLM call, so boundary detection is free.

Condition	Boundary precision	Boundary recall	F1	Documents evaluated
Clean text (no OCR noise)	1.000	1.000	1.000	660
Character-level OCR noise (2 %)	-	-	0.969	660

The eval set mixes multi-invoice PDFs with single-invoice negatives. Precision and recall are computed on exact page-index boundary matches. The 2 %% OCR noise model applies per-character confusion, dropping, and swapping (including colon loss, which breaks label regexes). At that noise level the detector achieves F1 0.969, which motivated the suggest-and-confirm UX: boundaries are offered to the user for review rather than applied silently.

Methodology Notes

All numbers are sourced from committed evaluation artifacts in the repository (docs/ocr-calibration/*.json, docs/OCR_IMAGE_CALIBRATION.md). Split-detection results are in docs/ocr-calibration/split_detection_results.json. No numbers are estimated or extrapolated.
OCR scoring uses order-insensitive multiset recall and precision: the metric counts whether each ground-truth word token appears in the OCR output (recall) and whether each OCR output token appears in the ground truth (precision), without penalizing for reading-order differences. This is a better proxy for field-extraction accuracy than sequence-similarity metrics, which are sensitive to line segmentation changes that do not affect token content.
Camera-photo ground truth is a human transcription of the test document (docs/IMG_2375.HEIC). Printed-page ground truth is derived from pdftotext, the same tool used in the digital-PDF fast path.
Split-detection precision and recall measure exact page-index matches on synthetically generated multi-invoice PDFs with known boundaries. The eval set covers a range of invoice types, date formats, and vendor names.
Benchmark re-runs after any engine change can be triggered with the scripts in scripts/. Fresh results land in dev-artifacts/ (gitignored) and are committed after review.

Questions about the methodology or results: [email protected]

Not measured yet

Two numbers a buyer will ask for are deliberately absent from this page, because no evaluation backs them:

Field-level extraction accuracy (precision, recall, F1 or exact match per field such as invoice number, total or tax ID). The OCR numbers above measure whether the words are read, not whether the right value lands in the right field. A labelled evaluation set is being assembled; until it exists, no percentage is published.
End-to-end latency (p50 and p95 from upload to result). Per-stage timings are recorded as metrics, but no baseline has been published.

Where DocSolved is known to do worse: camera photos of glossy or curved pages (word recall 0.865 against 1.000 on a clean printed page), numbers on that same photo (numeric recall 0.000, unchanged from the previous engine, while a detection threshold we do not ship reaches 1.000 on it), Polish diacritics on photos (0.775), and photo unwarping applied to an already flat page (0.635, which is why it runs on camera photos only). These cases can be reviewed rather than trusted: on a scanned document an extracted field carries the grounding the matcher could establish — matched source text, a match status, and a page location when the value is anchored — and may carry a confidence score. Structured e-invoice fields skip OCR entirely; they come from the source data and do not need those signals.

Machine-readable numbers: docs/benchmarks/results.json in the repository, derived from the committed artifacts, and the methodology in docs/benchmarks/METHODOLOGY.md.