docling-project / docling-project/docling

Docling error on memory allocation

Open
#1,126 7 comments 0 reactions 1 assignee Claimed by @vku-ibm View on GitHub
bug
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

### Bug

We are parsing multiple documents in parallel using dataflow. With CPU we didn't observe any problem, since we switched to GPU we got the following error:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 252.00 MiB. GPU 0 has a total capacity of 14.58 GiB of which 39.62 MiB is free. Process 165 has 0 bytes memory in use. Process 157 has 0 bytes memory in use. Process 168 has 0 bytes memory in use. Process 170 has 0 bytes memory in use. Of the allocated memory 2.44 GiB is allocated by PyTorch, and 368.84 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

This is our accelerator configuration:
accelerator_options = AcceleratorOptions(
num_threads=4, device=AcceleratorDevice.AUTO
)

I understand that we can improve memory allocation, but should docling wait until is able to allocate memory rather than fail?

### Steps to reproduce

Launch multiple parsing on GPU with
accelerator_options = AcceleratorOptions(
num_threads=4, device=AcceleratorDevice.AUTO
)

### Docling version

Docling version: 2.16.0
Docling Core version: 2.17.2
Docling IBM Models version: 3.3.1
Docling Parse version: 3.3.0
### Python version

Python 3.11.5

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.