docling-project / docling-project/docling

Vector-dense pages (CAD/schematic path storms): word-cell decode runs unconditionally in preprocessing and costs GiBs per page — request config surface to bound/skip it

Open
#4,058 12 comments 0 reactions 1 assignee Claimed by @PeterStaar-IBM View on GitHub
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

### Bug / feature request

Pages consisting of dense vector line-art (CAD drawings, wiring schematics, "path storms" of 100k+ line segments) make `get_segmented_page()` — the word-level cell decode in docling-parse — allocate **multiple GiB for a single page**. On docling 2.121 this decode runs **unconditionally in the preprocessing stage** (`PagePreprocessingModel._parse_page_cells`), so it cannot be avoided with `do_table_structure=False` or any other pipeline option. We are asking for a **config surface to bound or skip word-cell extraction on pathological pages**.

This is fully reproducible with a synthetic PDF (generator below — programmatic line segments, no real-world content).

### Measurements (synthetic, single A4 page, this exact call)

Instrumented `DoclingParsePageBackend.get_segmented_page` with an RSS high-water probe, `StandardPdfPipeline` with `do_ocr=False`; container pinned to 4 CPUs, linux/amd64:

| line segments on the page | decode wall time | RSS growth across the call | process peak RSS |
|---:|---:|---:|---:|
| 100k | 0.8 s | +0.26 GiB | 1.15 GiB |
| 250k | 2.2 s | +0.60 GiB | 1.44 GiB |
| 500k | 4.4 s | +1.2 GiB | 2.13 GiB |
| 1M | 8.9 s | +2.4 GiB | 3.49 GiB |

Scaling is linear at roughly **2.4 GiB per million segments**. The call site is identical with `do_table_structure=True` and `False`:

```
_parse_page_cells (page_preprocessing_model.py:75)
<- __call__ (page_preprocessing_model.py:51)
<- _process_batch (standard_pdf_pipeline.py:516)
<- _run (standard_pdf_pipeline.py:288)
```

### Real-world impact

We run a production document-processing fleet. Operations manuals (~400–450 pages) containing CAD/wiring-schematic pages with **100k–660k vector paths per page** (text drawn as paths; near-zero raster content, normal A4 mediabox) reproducibly OOM 12 GiB pods:

- a single 660k-path page costs **~5.5 GiB** on 2.121 (~4.6 GiB on 2.108, i.e. the 2.108→2.121 bumps of docling-parse 7.4→7.14 / pypdfium2 4.30→5.13 made it ~19 % worse);
- heavy pages cluster, so a 40-page window of one document needs **12.1 GiB** by itself;
- `release_native_memory_every_n_pages=4` helps only marginally there (~0.15 GiB on the worst window, byte-identical output);
- these documents fail on every docling version we tested; they are simply not ingestable under a bounded memory budget today.

On these pages the decode buys nothing: the layout model tends to classify the line grids as false-positive TABLE clusters (they are diagrams), and where text is drawn as vector paths there are no native word cells to find anyway — the existing empty-cells fallback in the table-structure stage produces the same output.

### Ask

A supported way to bound this, any of:

1. Expose docling-parse's `ContentConfig` word-cell level (e.g. `word_cells_content_level=SKIP`) through the backend/pipeline options, so an application can construct the backend with `create_words=False` semantics. The table-structure stage's existing empty-cells fallback already handles the "no segmented page / no word cells" case gracefully.
2. Or: a per-page guard — skip/degrade the word-level decode above a configurable path/char-count threshold (the pathological pages are 30–200× above any real data-table page we have measured; a dense ruled 100×12 grid is ~1.5–3k segments).
3. Or: document an intended supported path if one already exists.

We are happy to contribute a PR for (1) or (2) if maintainers agree on the shape.

Related (same "native memory on large/pathological PDFs" family, different triggers): #3671, #3345, docling-parse#227.

### Reproduction

make_pathstorm.py — synthetic path-storm generator (stdlib only, deterministic, no real-world data)

```python
#!/usr/bin/env python3
"""Generate a synthetic 'path-storm' PDF: A4 page(s) whose content is a dense
ruled grid drawn as N short vector line segments (no text, no images).
Mimics CAD/wiring-schematic exports where drawings are pure vector paths.
Usage: make_pathstorm.py [segments=500000] [out.pdf] [pages=1]"""
import sys
import zlib

W, H = 595, 842 # A4 points
TARGET = int(sys.argv[1]) if len(sys.argv) > 1 else 500_000
out = sys.argv[2] if len(sys.argv) > 2 else "pathstorm.pdf"
PAGES = int(sys.argv[3]) if len(sys.argv) > 3 else 1 # pages share one content stream

ops = ["0.4 w"]
n = 0
rows = int((TARGET / 2) ** 0.5 * (H / W) ** 0.5)
cols_per_row = (TARGET // 2) // rows
dash = W / cols_per_row / 1.6
for r in range(rows):
y = 20 + (H - 40) * r / rows
for c in range(cols_per_row):
x = 15 + (W - 30) * c / cols_per_row
ops.append(f"{x:.1f} {y:.1f} m {x + dash:.1f} {y:.1f} l S")
n += 1
cols = rows * W // H
rows_per_col = (TARGET // 2) // cols
dash_v = H / rows_per_col / 1.6
for c in range(cols):
x = 20 + (W - 40) * c / cols
for r in range(rows_per_col):
y = 15 + (H - 30) * r / rows_per_col
ops.append(f"{x:.1f} {y:.1f} m {x:.1f} {y + dash_v:.1f} l S")
n += 1

stream = zlib.compress("\n".join(ops).encode())
content_obj = 3 + PAGES
kids = " ".join(f"{3+i} 0 R" for i in range(PAGES))
objs = []
objs.append(b"<< /Type /Catalog /Pages 2 0 R >>")
objs.append(f"<< /Type /Pages /Kids [{kids}] /Count {PAGES} >>".encode())
for i in range(PAGES):
objs.append(
f"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 {W} {H}] /Contents {content_obj} 0 R /Resources << >> >>".encode()
)
objs.append(
f"<< /Length {len(stream)} /Filter /FlateDecode >>\nstream\n".encode()
+ stream
+ b"\nendstream"
)

pdf = bytearray(b"%PDF-1.4\n")
offsets = []
for i, body in enumerate(objs, 1):
offsets.append(len(pdf))
pdf += f"{i} 0 obj\n".encode() + body + b"\nendobj\n"
xref = len(pdf)
pdf += f"xref\n0 {len(objs)+1}\n0000000000 65535 f \n".encode()
for off in offsets:
pdf += f"{off:010d} 00000 n \n".encode()
pdf += (
f"trailer\n<< /Size {len(objs)+1} /Root 1 0 R >>\nstartxref\n{xref}\n%%EOF".encode()
)
open(out, "wb").write(bytes(pdf))
print(f"{out}: {PAGES} page(s) x {n} line segments, {len(pdf)/1e6:.2f} MB")
```

```bash
python make_pathstorm.py 500000 storm.pdf # 2.6 MB, one A4 page, ~500k segments

python - <<'EOF'
from docling.datamodel.base_models import InputFormat
from docling.datamodel.pipeline_options import PdfPipelineOptions
from docling.document_converter import DocumentConverter, PdfFormatOption
import resource
opts = PdfPipelineOptions(); opts.do_ocr = False; opts.do_table_structure = False
conv = DocumentConverter(format_options={InputFormat.PDF: PdfFormatOption(pipeline_options=opts)})
res = conv.convert("storm.pdf")
print(res.status, "peak RSS GiB:", resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024 / 1024)
EOF
# -> success peak RSS ~2.1 GiB for one near-empty-looking page, tables and OCR both OFF
```

### Docling version

```
Docling version: 2.121.0
Docling Core version: 2.92.0
Docling IBM Models version: 3.14.0
Docling Parse version: 7.14.0
pypdfium2: 5.13.0
Python: 3.10.18, linux/amd64 (container, 4 CPUs)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.