docling-project / docling-project/docling

Docling return nothing when using VLM pipeline

Open
#3,033 14 comments 1 reaction 0 assignees View on GitHub
bug triage/close-fixed
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
3d 4h
Merged PRs (30d)
95

Description

### Bug

Docling return nothing when I run the VLM pipeline of Docling. I try to convert the example PDF by using Docling VLM pipeline. Here is the PDF: [Link](https://www.w3.org/WAI/WCAG20/Techniques/working-examples/PDF20/table.pdf)

### Steps to reproduce

Here is my Python code, I used VLLM to serve `ibm-granite/granite-docling-258M.`

```
import datetime
import logging
import time
from pathlib import Path

import numpy as np
from pydantic import TypeAdapter

from docling.datamodel.base_models import ConversionStatus, InputFormat
from docling.datamodel.pipeline_options import (
VlmConvertOptions,
VlmPipelineOptions,
)
from docling.datamodel.settings import settings
from docling.datamodel.vlm_engine_options import (
ApiVlmEngineOptions,
VlmEngineType,
)
from docling.document_converter import DocumentConverter, PdfFormatOption
from docling.pipeline.vlm_pipeline import VlmPipeline
from docling.utils.profiling import ProfilingItem

_log = logging.getLogger(__name__)


def main():
logging.getLogger("docling").setLevel(logging.WARNING)
_log.setLevel(logging.INFO)

BATCH_SIZE = 64

settings.perf.page_batch_size = BATCH_SIZE
settings.debug.profile_pipeline_timings = True

data_folder = Path(__file__).parent / "../../tests/data"
# input_doc_path = data_folder / "pdf" / "2305.03393v1.pdf" # 14 pages
input_doc_path = data_folder / "pdf" / "table.pdf" # 18 pages

# Use the granite_docling preset with API runtime override for vLLM
vlm_options = VlmConvertOptions.from_preset(
"granite_docling",
engine_options=ApiVlmEngineOptions(
runtime_type=VlmEngineType.API,
url="http://localhost:8000/v1/chat/completions",
concurrency=BATCH_SIZE,
),
)

pipeline_options = VlmPipelineOptions(
vlm_options=vlm_options,
enable_remote_services=True, # required when using a remote inference service.
)

doc_converter = DocumentConverter(
format_options={
InputFormat.PDF: PdfFormatOption(
pipeline_cls=VlmPipeline,
pipeline_options=pipeline_options,
),
}
)

start_time = time.time()
doc_converter.initialize_pipeline(InputFormat.PDF)
end_time = time.time() - start_time
_log.info(f"Pipeline initialized in {end_time:.2f} seconds.")

now = datetime.datetime.now()
conv_result = doc_converter.convert(input_doc_path)
assert conv_result.status == ConversionStatus.SUCCESS

num_pages = len(conv_result.pages)
pipeline_runtime = conv_result.timings["pipeline_total"].times[0]
_log.info(f"Document converted in {pipeline_runtime:.2f} seconds.")
_log.info(f" [efficiency]: {num_pages / pipeline_runtime:.2f} pages/second.")
for stage in ("page_init", "vlm"):
values = np.array(conv_result.timings[stage].times)
_log.info(
f" [{stage}]: {np.min(values):.2f} / {np.median(values):.2f} / {np.max(values):.2f} seconds/page"
)

TimingsT = TypeAdapter(dict[str, ProfilingItem])
timings_file = Path(f"result-timings-gpu-vlm-{now:%Y-%m-%d_%H-%M-%S}.json")
with timings_file.open("wb") as fp:
r = TimingsT.dump_json(conv_result.timings, indent=2)
fp.write(r)
_log.info(f"Profile details in {timings_file}.")


if __name__ == "__main__":
main()
```

Result of `conv_result.document.export_to_markdown()` will be empty string. `tables` and `texts` of `conv_result.document` are also empty list.

### Docling version

Docling v2.74.0

### Python version

Python 3.10.12

Contributor guide

Open the contributing guide

Research direction

Start with the VlmPipeline configuration shown in the reproduction, including VlmConvertOptions, ApiVlmEngineOptions, and the DocumentConverter PDF format option. Run the example against the linked PDF and vLLM endpoint, then inspect the conversion result and pipeline timings. Done means the VLM pipeline produces populated text and table data and export_to_markdown() is no longer empty.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.