docling-project / docling-project/docling

docling-serve OOM during do_chart_extraction = True on 16gb T4 because of default images_scaling = 2

Open
#3,216 1 comment 0 reactions 0 assignees View on GitHub
bug triage/close-stale
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

### Bug
docling-serve running OOM on GPU T4 (16gb VRAM) when running do_chart_extraction = True (and do_picture_classification = True) on an image / pdf page where the underlying chart takes up a very large propotion of the page/image. Docling is attempting to get allocated an additional 4gb of VRAM during the enrichment step causing OOM. This does **not** happen when there are multiple charts that take up a large, only when there is 1 'disproportionally large' chart. The reasons seems to be that the chart that is being processed is resized by 2x because of default setting images_scale = 2 (i.e. area is 4x bigger). The granite-vision model's generate() method (called at line 204 of granite_vision.py) seems to need to allocate GPU memory proportional to the number of 384x384 image tiles, and large chart regions at images_scale=2.0 produce too many tiles (9+) to fit in the remaining ~3 GB of T4 VRAM after the 11.4 GB model is loaded.

### Steps to reproduce
Pass a image where the chart takes up a _large_ area into docling-serve with do_chart_extraction = True and do_picture_classification = True. Image with extracted chart sized 668x367 was processed with images_scaling = 1 but not with images_scaling = 2.

Code:

Bash(DOCLING_URL="https://{url}.eastus.azurecontainerapps.io"
RESULT=$(curl -s -X POST "$DOCLING_URL/v1/convert/file/async" \
-F "files=@{filename}.pdf" \
-F "to_formats=md" \
-F "do_chart_extraction=true" \
-F "do_picture_classification=true" \
-F "include_images=false" 2>&1)

However, the bug does not occur when scaling_images = 1.0 (vs. if let unset defaults to 2.0).

### Docling version
docling-serve==1.15.1

### Python version
python 3.11

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.