CUDA driver error: invalid argumen

### Bug
I am running Docling PdfPipeline on an A100 GPU.
When running my code, I get:

```
2025-10-31 14:04:57,932 - INFO - detected formats: []
2025-10-31 14:04:57,985 - INFO - Going to convert document batch...
2025-10-31 14:04:57,985 - INFO - Initializing pipeline for StandardPdfPipeline with options hash 2e26134dac131144f27cb7dd96185dd5
2025-10-31 14:04:57,993 - INFO - Loading plugin 'langchain_docling'
2025-10-31 14:04:58,037 - INFO - Loading plugin 'docling_defaults'
2025-10-31 14:04:58,037 - INFO - Registered picture descriptions: ['langchain', 'vlm', 'api']
2025-10-31 14:04:58,388 - INFO - Accelerator device: 'cuda:0'
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████| 2/2 [00:02<00:00, 1.18s/it]
2025-10-31 14:05:03,194 - INFO - Loading plugin 'docling_defaults'
2025-10-31 14:05:03,196 - INFO - Registered ocr engines: ['auto', 'easyocr', 'ocrmac', 'rapidocr', 'tesserocr', 'tesseract']
2025-10-31 14:05:03,470 - INFO - Accelerator device: 'cuda:0'
[INFO] 2025-10-31 14:05:03,497 [RapidOCR] base.py:22: Using engine_name: torch
[INFO] 2025-10-31 14:05:03,522 [RapidOCR] download_file.py:60: File exists and is valid: /bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/rapidocr/models/ch_PP-OCRv4_det_infer.pth
[INFO] 2025-10-31 14:05:03,522 [RapidOCR] torch.py:54: Using /bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/rapidocr/models/ch_PP-OCRv4_det_infer.pth
[INFO] 2025-10-31 14:05:03,967 [RapidOCR] base.py:22: Using engine_name: torch
[INFO] 2025-10-31 14:05:03,969 [RapidOCR] download_file.py:60: File exists and is valid: /bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/rapidocr/models/ch_ptocr_mobile_v2.0_cls_infer.pth
[INFO] 2025-10-31 14:05:03,969 [RapidOCR] torch.py:54: Using /bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/rapidocr/models/ch_ptocr_mobile_v2.0_cls_infer.pth
[INFO] 2025-10-31 14:05:04,319 [RapidOCR] base.py:22: Using engine_name: torch
[INFO] 2025-10-31 14:05:04,364 [RapidOCR] download_file.py:60: File exists and is valid: /bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/rapidocr/models/ch_PP-OCRv4_rec_infer.pth
[INFO] 2025-10-31 14:05:04,365 [RapidOCR] torch.py:54: Using /bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/rapidocr/models/ch_PP-OCRv4_rec_infer.pth
2025-10-31 14:05:04,734 - INFO - Accelerator device: 'cuda:0'
2025-10-31 14:05:06,008 - INFO - Accelerator device: 'cuda:0'
2025-10-31 14:05:06,398 - INFO - Processing document Overheating Assessment.pdf
2025-10-31 14:05:33,322 - WARNING - Encountered an error during conversion of document 6ad38b46c94c803b4ec7e421c4f70631a7a4a4177d10bd85454c93a3ab2e8e5f:
Traceback (most recent call last):

File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/docling/pipeline/base_pipeline.py", line 230, in _build_document
for p in pipeline_pages: # Must exhaust!

File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/docling/pipeline/base_pipeline.py", line 195, in _apply_on_pages
yield from page_batch

File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/docling/models/page_assemble_model.py", line 70, in __call__
for page in page_batch:

File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/docling/models/table_structure_model.py", line 177, in __call__
for page in page_batch:

File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/docling/models/layout_model.py", line 174, in __call__
batch_predictions = self.layout_predictor.predict_batch( # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^

File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/docling_ibm_models/layoutmodel/layout_predictor.py", line 222, in predict_batch
outputs = self._model(**inputs)
^^^^^^^^^^^^^^^^^^^^^

File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/transformers/models/rt_detr_v2/modeling_rt_detr_v2.py", line 1920, in forward
outputs = self.model(
^^^^^^^^^^^

File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/transformers/models/rt_detr_v2/modeling_rt_detr_v2.py", line 1587, in forward
level_start_index = torch.cat((spatial_shapes.new_zeros((1,)), spatial_shapes.prod(1).cumsum(0)[:-1]))
^^^^^^^^^^^^^^^^^^^^^^

RuntimeError: CUDA driver error: invalid argument

Traceback (most recent call last):
File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/t0-bsr/pdf-parsing/load_docs.py", line 79, in
for doc in loader.lazy_load():
File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/langchain_docling/loader.py", line 117, in lazy_load
conv_res = self._converter.convert(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/pydantic/_internal/_validate_call.py", line 39, in wrapper_function
return wrapper(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/pydantic/_internal/_validate_call.py", line 136, in __call__
res = self.__pydantic_validator__.validate_python(pydantic_core.ArgsKwargs(args, kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/docling/document_converter.py", line 237, in convert
return next(all_res)
^^^^^^^^^^^^^
File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/docling/document_converter.py", line 260, in convert_all
for conv_res in conv_res_iter:
File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/docling/document_converter.py", line 332, in _convert
for item in map(
File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/docling/document_converter.py", line 379, in _process_document
conv_res = self._execute_pipeline(in_doc, raises_on_error=raises_on_error)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/docling/document_converter.py", line 402, in _execute_pipeline
conv_res = pipeline.execute(in_doc, raises_on_error=raises_on_error)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/docling/pipeline/base_pipeline.py", line 80, in execute
raise e
File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/docling/pipeline/base_pipeline.py", line 72, in execute
conv_res = self._build_document(conv_res)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/docling/pipeline/base_pipeline.py", line 270, in _build_document
raise e
File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/docling/pipeline/base_pipeline.py", line 230, in _build_document
for p in pipeline_pages: # Must exhaust!
File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/docling/pipeline/base_pipeline.py", line 195, in _apply_on_pages
yield from page_batch
File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/docling/models/page_assemble_model.py", line 70, in __call__
for page in page_batch:
File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/docling/models/table_structure_model.py", line 177, in __call__
for page in page_batch:
File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/docling/models/layout_model.py", line 174, in __call__
batch_predictions = self.layout_predictor.predict_batch( # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/docling_ibm_models/layoutmodel/layout_predictor.py", line 222, in predict_batch
outputs = self._model(**inputs)
^^^^^^^^^^^^^^^^^^^^^
File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/transformers/models/rt_detr_v2/modeling_rt_detr_v2.py", line 1920, in forward
outputs = self.model(
^^^^^^^^^^^
File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/bask/projects/u/usjs9456-ati-test/rwood/t0_bsr/venv_pdf/lib/python3.11/site-packages/transformers/models/rt_detr_v2/modeling_rt_detr_v2.py", line 1587, in forward
level_start_index = torch.cat((spatial_shapes.new_zeros((1,)), spatial_shapes.prod(1).cumsum(0)[:-1]))
^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA driver error: invalid argument

```

### Steps to reproduce
I am using CUDA 12.6, PyTorch 2.8.0 and docling 2.59.0.

Here is my code:

```
import os
import time
import pickle
from glob import glob

from langchain_docling import DoclingLoader

from docling.datamodel import vlm_model_specs
from docling.datamodel.base_models import InputFormat
from docling.datamodel.pipeline_options import (
VlmPipelineOptions,
PdfPipelineOptions,
RapidOcrOptions,
granite_picture_description,
)
from docling.document_converter import (
DocumentConverter,
ImageFormatOption,
PdfFormatOption,
)
from docling.pipeline.standard_pdf_pipeline import StandardPdfPipeline
from docling.pipeline.vlm_pipeline import VlmPipeline

IMAGE_RESOLUTION_SCALE = 1.0

def get_converter(model=vlm_model_specs.SMOLDOCLING_TRANSFORMERS):
"""Run Docling.

Parameters
----------
model : str, optional
Model to use, by default vlm_model_specs.SMOLDOCLING_TRANSFORMERS
"""
# pdf pipeline options
pdf_pipeline_options = PdfPipelineOptions(enable_remote_services=True)
pdf_pipeline_options.allow_external_plugins = True
pdf_pipeline_options.ocr_options = RapidOcrOptions(lang=["english"], backend="torch")
pdf_pipeline_options.images_scale = IMAGE_RESOLUTION_SCALE
pdf_pipeline_options.generate_page_images = False
pdf_pipeline_options.generate_picture_images = True
pdf_pipeline_options.do_picture_description = True
pdf_pipeline_options.picture_description_options = granite_picture_description

# vlm pipeline options
vlm_pipeline_options = VlmPipelineOptions(enable_remote_services=True)
vlm_pipeline_options.allow_external_plugins = True
vlm_pipeline_options.images_scale = IMAGE_RESOLUTION_SCALE
vlm_pipeline_options.generate_page_images = False
vlm_pipeline_options.generate_picture_images = True
vlm_pipeline_options.vlm_options = model
vlm_pipeline_options.do_picture_description = True
vlm_pipeline_options.picture_description_options = granite_picture_description

return DocumentConverter(
format_options={
InputFormat.IMAGE: ImageFormatOption(
pipeline_options=vlm_pipeline_options,
pipeline_cls=VlmPipeline,
),
InputFormat.PDF: PdfFormatOption(
pipeline_options=pdf_pipeline_options,
pipeline_cls=StandardPdfPipeline,
),
}
)

if os.path.exists("./data/docs.pkl"):
with open("./data/docs.pkl", "rb") as f:
docs = pickle.load(f)
else:
loader = DoclingLoader(
file_path=glob("./data/downloaded-content/*.pdf"),
converter=get_converter(),
# convert_kwargs={"raises_on_error":False}
)
docs = []
for doc in loader.lazy_load():
try:
docs.append(doc)
except Exception as e:
print(f"Failed to process document: {e}")

with open("./data/docs.pkl", "wb") as f:
pickle.dump(docs, f, protocol=pickle.HIGHEST_PROTOCOL)

```

### Docling version

```
(venv_pdf) [oeeo2350@bask-pg0309u04a pdf-parsing]$ docling --version
2025-10-31 14:07:20,890 - WARNING - The plugin langchain_docling will not be loaded because Docling is being executed with allow_external_plugins=false.
2025-10-31 14:07:20,890 - INFO - Loading plugin 'docling_defaults'
2025-10-31 14:07:20,892 - INFO - Registered ocr engines: ['auto', 'easyocr', 'ocrmac', 'rapidocr', 'tesserocr', 'tesseract']
Docling version: 2.59.0
Docling Core version: 2.50.0
Docling IBM Models version: 3.10.2
Docling Parse version: 4.7.0
Python: cpython-311 (3.11.3)
Platform: Linux-4.18.0-553.el8_10.x86_64-x86_64-with-glibc2.28
```

### Python version
Python 3.11.3

Full `pip list`

```
Package Version
------------------------- ------------
accelerate 1.11.0
aiohappyeyeballs 2.6.1
aiohttp 3.13.2
aiosignal 1.4.0
annotated-types 0.7.0
antlr4-python3-runtime 4.9.3
anyio 4.11.0
asttokens 3.0.0
attrs 25.4.0
beautifulsoup4 4.14.2
certifi 2025.10.5
charset-normalizer 3.4.4
click 8.3.0
colorlog 6.10.1
comm 0.2.3
contourpy 1.3.3
cycler 0.12.1
datasets 4.3.0
debugpy 1.8.17
decorator 5.2.1
dill 0.4.0
docling 2.59.0
docling-core 2.50.0
docling-ibm-models 3.10.2
docling-parse 4.7.0
et_xmlfile 2.0.0
executing 2.2.1
Faker 37.12.0
filelock 3.20.0
filetype 1.2.0
fonttools 4.60.1
frozenlist 1.8.0
fsspec 2025.9.0
h11 0.16.0
hf-xet 1.2.0
httpcore 1.0.9
httpx 0.28.1
huggingface-hub 0.36.0
idna 3.11
inquirerpy 0.3.4
ipykernel 7.1.0
ipython 9.6.0
ipython_pygments_lexers 1.1.1
jedi 0.19.2
Jinja2 3.1.6
jsonlines 4.0.0
jsonpatch 1.33
jsonpointer 3.0.0
jsonref 1.1.0
jsonschema 4.25.1
jsonschema-specifications 2025.9.1
jupyter_client 8.6.3
jupyter_core 5.9.1
kiwisolver 1.4.9
langchain-core 0.3.79
langchain-docling 1.1.0
langsmith 0.4.38
latex2mathml 3.78.1
lxml 6.0.2
markdown-it-py 4.0.0
marko 2.2.1
MarkupSafe 3.0.3
matplotlib 3.10.7
matplotlib-inline 0.2.1
mdurl 0.1.2
mpire 2.10.2
mpmath 1.3.0
multidict 6.7.0
multiprocess 0.70.16
nest-asyncio 1.6.0
networkx 3.5
numpy 2.2.6
nvidia-cublas-cu12 12.6.4.1
nvidia-cuda-cupti-cu12 12.6.80
nvidia-cuda-nvrtc-cu12 12.6.77
nvidia-cuda-runtime-cu12 12.6.77
nvidia-cudnn-cu12 9.10.2.21
nvidia-cufft-cu12 11.3.0.4
nvidia-cufile-cu12 1.11.1.6
nvidia-curand-cu12 10.3.7.77
nvidia-cusolver-cu12 11.7.1.2
nvidia-cusparse-cu12 12.5.4.2
nvidia-cusparselt-cu12 0.7.1
nvidia-nccl-cu12 2.27.3
nvidia-nvjitlink-cu12 12.6.85
nvidia-nvshmem-cu12 3.3.20
nvidia-nvtx-cu12 12.6.77
omegaconf 2.3.0
opencv-python 4.12.0.88
openpyxl 3.1.5
orjson 3.11.4
packaging 25.0
pandas 2.3.3
parso 0.8.5
pexpect 4.9.0
pfzy 0.3.4
pillow 11.3.0
pip 22.3.1
platformdirs 4.5.0
pluggy 1.6.0
polyfactory 2.22.3
prompt_toolkit 3.0.52
propcache 0.4.1
psutil 7.1.2
ptyprocess 0.7.0
pure_eval 0.2.3
pyarrow 22.0.0
pyclipper 1.3.0.post6
pydantic 2.12.3
pydantic_core 2.41.4
pydantic-settings 2.11.0
Pygments 2.19.2
pylatexenc 2.10
pyparsing 3.2.5
pypdfium2 4.30.0
python-dateutil 2.9.0.post0
python-docx 1.2.0
python-dotenv 1.2.1
python-pptx 1.0.2
pytz 2025.2
PyYAML 6.0.3
pyzmq 27.1.0
rapidocr 3.4.2
referencing 0.37.0
regex 2025.10.23
requests 2.32.5
requests-toolbelt 1.0.0
rich 14.2.0
rpds-py 0.28.0
rtree 1.4.1
safetensors 0.6.2
scipy 1.16.3
semchunk 2.2.2
setuptools 65.5.0
shapely 2.1.2
shellingham 1.5.4
six 1.17.0
sniffio 1.3.1
soupsieve 2.8
stack-data 0.6.3
sympy 1.14.0
tabulate 0.9.0
tenacity 9.1.2
tokenizers 0.22.1
torch 2.8.0+cu126
torchaudio 2.8.0+cu126
torchvision 0.23.0+cu126
tornado 6.5.2
tqdm 4.67.1
traitlets 5.14.3
transformers 4.57.1
triton 3.4.0
typer 0.19.2
typing_extensions 4.15.0
typing-inspection 0.4.2
tzdata 2025.2
urllib3 2.5.0
wcwidth 0.2.14
xlsxwriter 3.2.9
xxhash 3.6.0
yarl 1.22.0
zstandard 0.25.0
```