docling-project / docling-project/docling-parse

Bound the native content-stream parse on vector-dense pages (path-storm PDFs cost GiBs per page in `get_page()`)

Open
#345 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
333
Forks
80
Avg merge
1d 14h
Merged PRs (30d)
10

Description

## Summary

`DoclingPdfParser.get_page()` parses and materialises every path in a page's content stream before any `ContentConfig` level or `create_words`/`create_textlines` flag can bound the work. On CAD / wiring-schematic pages drawn as hundreds of thousands of vector paths this costs multiple GiB of native memory per page, and there is no configuration that limits it. Downstream this is the OOM class tracked in docling-project/docling#4058.

## Measurements

Synthetic path-storm page (dense ruled grid of N line segments, generator in docling#4058), Docling 2.121 / docling-parse 7.14.0, amd64, 4 CPU:

| segments | peak RSS |
|---|---|
| 100k | ~0.6 GiB |
| 1M | 3.49 GiB (default backend, tables on or off identical) |
| 1M, all `ContentLevel` SKIP + `compute_shapes=False` | 2.75 GiB — the parse itself dominates, not materialisation |
| 1M, pypdfium2 backend | 1.42 GiB, 3× faster |

Real documents (equipment manuals, 337–480 pages, worst pages 466k–970k paths, 12 GiB cgroup, 40-page batches, Docling 2.121/2.122 with docling-parse 7.14.0):

| document | docling-parse default | `OcrMode.FULL_PAGE` (decode skipped since docling#4061) | pypdfium2 backend |
|---|---|---|---|
| 352 pp, worst page 970k paths | OOMKilled at 12 GiB | 6.55 GiB peak | 6.16 GiB peak |
| 337 pp, worst page 660k paths | OOMKilled at 12 GiB (2.41) | 6.19 GiB | 6.03 GiB |
| 480 pp, worst page 466k paths | 11.34 GiB peak, no headroom | 7.25 GiB | 4.98 GiB |

Full details and the docling-side discussion: docling-project/docling#4058 (this issue is the docling-parse half that thread left open).

## Ask

A per-page budget or degradation threshold inside `get_page()` — for example a maximum path/segment count (or a byte budget on the decoded content stream) above which the parser stops materialising paths for that page and reports it as degraded, so callers can fall back (OCR-only, or a different backend) without the process being killed first. The docling side is already able to consume a `parsed_page=None`-style outcome since docling#4061.

Happy to contribute the change if maintainers can point at the preferred place for such a budget in the C++ parser.

Contributor guide

Open the contributing guide

Research direction

Start at DoclingPdfParser.get_page() and trace the native C++ content-stream parse and path materialisation. Compare the existing ContentConfig, ContentLevel, and compute_shapes handling, then define a per-page path or byte budget that yields the parsed_page=None-style degraded outcome for docling#4061 to consume. Done means path-storm pages avoid OOM before materialisation.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.