docling-project / docling-project/docling-core

Feature proposal: per-item OCR confidence on document-level TextItem and TableCell

Open
#732 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
HTML
Stars
282
Forks
214
Avg merge
2d 15h
Merged PRs (30d)
21

Description

**Requested feature**

An optional `confidence: float | None` (default `None`, `ge=0, le=1`, `CONFID_PREC` serializer, matching `BasePrediction`) on document-level `TextItem` and `TableCell`, so consumers can route low-confidence lines to a correction step (cf. docling [#2711](https://github.com/docling-project/docling/issues/2711)) or filter table cells without re-running OCR.

**Current state**

`BasePrediction` carries confidence into several `meta` fields, and the page-level `TableCell` has `confidence: float = 1.0` with `from_ocr`. The document-level items have nothing; docling's `ConfidenceReport` aggregates per page but does not expose per item.

**Why a first-class field rather than `meta`**

`TableData.table_cells` are plain models with no `meta` field, so per-cell confidence has no home there. For `TextItem`, a first-class field is queryable and projector-aware, and can propagate through the chunkers rather than depending on namespaced custom fields.

**Semantics**

Recognition confidence from the producing engine. `None` keeps absence distinguishable from digital-born text (unlike the page-level `1.0` convention, since document-level items mix both sources).

**Versioning**

Happy to ship the full #703 regime with the PR: `CURRENT_VERSION` bump, a registered downgrade projector that strips the fields, schema snapshot and tests.

**Alternatives considered**

- Extending `ProvenanceItem`: rejected, recognition confidence is a property of the item, not of one provenance entry, and multiple `prov` entries would make it ambiguous.
- Keeping aggregation only in docling's `ConfidenceReport`: rejected, per-item routing is exactly what an aggregate cannot express.

Direction agreed, I'll follow with a PR. For context, I'm also working on table structure correctness in docling ([#4088](https://github.com/docling-project/docling/pull/4088)).

Contributor guide

Open the contributing guide

Research direction

Start by tracing BasePrediction, the document-level TextItem and TableCell models, and TableData.table_cells to understand the existing confidence conventions. The work is complete when both items expose the optional confidence field and the agreed #703 versioning regime includes a downgrade projector, schema snapshot, and tests.

Written by the indexing model from the issue text.

Assessment

Domain
backend-api-design
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.