docling-project / docling-project/docling

Add support for PP-DocLayout-V3 layout detection and polygon bounding boxes.

Open
#3,224 2 comments 6 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

### Requested feature
Add support for multi-point bounding boxes layout detection models such as [PP-DocLayout-V3](https://huggingface.co/PaddlePaddle/PP-DocLayoutV3_safetensors).

This allows docling to handle scewed and curved scans / images.
Moreover, PP-DocLayout-V3 proved to be a superior layout detection model than the current default.

The implementation would need the following changes:
* Extend `docling-core` with a Polygon type and basic geometric operations (IoU, containment)
* Add a polygon field to `docling.datamodel.base_models.Cluster`
* Update `OnnxRuntimeObjectDetectionEngine` to handle the additional mask / polygon output tensor from PP-DocLayout-V3. Maybe reuse `_safe_extract` logic from GLM-OCR's code base: https://github.com/zai-org/GLM-OCR/blob/b069c6f44fad65c35e0f2d3cf6ababba3e3f6d82/glmocr/layout/layout_detector.py#L84
* Update the `LayoutPostprocessor` to handle overlap resolution of polygon-to-polygon intersections and ensure that the reading order logic to work with polygons / PP-DocLayout-V3
* Polygon masking: Right now docling crops bounding boxes to images. With polygons we would get noisy crops possibly containing text from out of polygon regions. GLM-OCR handles this in their code base via masking: Instead of a simple `image.crop(bbox)` do a Axis-Aligned Bounding Box crop and apply a polygon mask to white out noise outside the segment.
* GLM-OCR is able to handle skewed images. GLM-OCR is supported since the latest docling version. For other OCR models some sort of perspective warp would need to be applied in order to improve results. This is optional and one could argue that handling skewed images for OCR are limitations of some OCR models.

### Alternatives
PP-DocLayout-V3 layout detection is already possible through doclings great plugin system. We published such a plugin here: https://github.com/DCC-BS/docling-pp-doc-layout
The downside of this approach is, that we do not gain the benefits of the multi-point bounding boxes. This change needs deep integration into doclings code base.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.