docling-project / docling-project/docling

Feature request: fallback table detection for picture-classified regions (try_table_on_picture)

Open
#3,410 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

## Summary

Add an option to run table structure detection on regions classified as `picture` by the
layout model, when those regions exhibit table-like characteristics (cell grid structure,
row/column alignment).

## Current behaviour

All three table structure models (TableFormer V1/V2, Granite Vision) only process regions
labelled `TABLE` or `DOCUMENT_INDEX`. If RT-DETR classifies a region as `PICTURE`, table
structure detection is never invoked — confirmed by Dosu in #[link your issue here].

Related issues: #1653, #767.

## Proposed option

```python
opts = PdfPipelineOptions()
opts.try_table_on_picture = True # attempt TableFormer on picture-classified regions
Motivation / use case
Industrial PDF manuals (e.g. operator manuals for crane systems) contain procedure tables
where each row has: number | text description | diagram/icon. RT-DETR consistently
classifies these as picture because of the embedded images per row.

Result: all checklist text is lost ( in output), making the document
unusable for RAG / information retrieval.

Current workarounds and their limitations
TableFormerMode.ACCURATE — no effect (region never reaches TableFormer)
do_picture_description=True with Granite Vision — produces descriptions, not structured table markdown; improves with v2.93.0 but still not a proper table structure
TableCropsLayoutModel — marks entire page as TABLE, too aggressive for mixed pages
Expected behaviour
When try_table_on_picture=True:

For each picture-classified region, attempt table structure detection
If TableFormer returns a valid table (confidence > threshold), use the table output
Otherwise fall back to picture description /
This is the minimal viable fix. The TableCropsLayoutModel experimental code already
demonstrates the concept is feasible within the existing architecture.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.