docling-project / docling-project/docling
VLM API call failures are silently converted to empty pages: ConversionStatus.SUCCESS, empty errors, and raises_on_error=True never fires
- Dominant language
- Python
- Stars
- 66.4k
- Forks
- 4.8k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 84
Description
When the remote VLM endpoint fails during page conversion in the VLM pipeline (read timeout, connection error, or a gateway error body that fails response parsing), the failure is swallowed and the affected pages come back **empty**, while the conversion reports **complete success**. There is no programmatic way to distinguish "this document is genuinely blank" from "every API call failed".
The swallow chain:
1. `api_image_request()` catches **every** exception, logs it, and returns empty text as a normal result ([api_image_request.py#L257-L259](https://github.com/docling-project/docling/blob/61d76f1ff3f8/docling/utils/api_image_request.py#L257-L259)):
```python
except Exception as e:
_log.error(f"Error, could not process request: {e}")
return ApiImageRequestResult("", 0, VlmStopReason.UNSPECIFIED)
```
2. The API inference engine wraps that into a normal `VlmEngineOutput`/`VlmPrediction` without checking text or stop reason.
3. `VlmPipeline._determine_status()` flags only `LENGTH` and `CONTENT_FILTERED` stop reasons (added in #3051) — `UNSPECIFIED`, the exact marker a swallowed exception produces, is not checked, so the status stays `SUCCESS` and `conv_res.errors` stays empty ([vlm_pipeline.py#L236-L268](https://github.com/docling-project/docling/blob/61d76f1ff3f8/docling/pipeline/vlm_pipeline.py#L236-L268)).
4. Even a downgraded status would not raise: `raises_on_error` only raises for statuses outside `{SUCCESS, PARTIAL_SUCCESS}` ([document_converter.py#L569-L581](https://github.com/docling-project/docling/blob/61d76f1ff3f8/docling/document_converter.py#L569-L581)).
The retry added in #3515 does not close this: `Retry(..., read=0, ...)` explicitly excludes read timeouts ([api_image_request.py#L29-L46](https://github.com/docling-project/docling/blob/61d76f1ff3f8/docling/utils/api_image_request.py#L29-L46)), the `backoff_factor=0.1` retries span only ~3 seconds (no help against an outage lasting minutes), and after exhaustion the same `except Exception` swallow applies.
The same transport-level swallow also affects `PictureDescriptionVlmEngineModel` (images silently get `""` descriptions), but the page-conversion case is the destructive one: the empty text *is* the page content.
**Real-world impact:** we run docling as the conversion step of a document-ingestion service feeding a RAG index. During a temporary outage of the gateway in front of our VLM endpoint, every per-page call hit the 180 s read timeout; multi-page PDFs "converted successfully" with every page blank, and the pipeline committed empty documents into the index — overwriting previously good content, since our flow deletes-then-reingests. Roughly a hundred documents were silently emptied in a 15-minute window, discovered only days later when retrieval returned nothing for them. Nothing in `ConversionResult` (status, errors, document) distinguished them from genuinely empty files; the only trace was ERROR lines on the `docling.utils.api_image_request` logger, which we now scrape with a `logging.Handler` as a workaround.
### Steps to reproduce
```python
from pypdf import PdfWriter
pdf_path = "/tmp/one_page.pdf"
w = PdfWriter()
w.add_blank_page(width=200, height=200)
with open(pdf_path, "wb") as f:
w.write(f)
from docling.datamodel.base_models import InputFormat
from docling.datamodel.pipeline_options import VlmConvertOptions, VlmPipelineOptions
from docling.datamodel.vlm_engine_options import ApiVlmEngineOptions, VlmEngineType
from docling.document_converter import DocumentConverter, PdfFormatOption
from docling.pipeline.vlm_pipeline import VlmPipeline
engine_options = ApiVlmEngineOptions(
runtime_type=VlmEngineType.API,
url="http://127.0.0.1:9/v1/chat/completions", # nothing listens here
concurrency=1,
timeout=5,
)
vlm_options = VlmConvertOptions.from_preset("granite_docling", engine_options=engine_options)
pipeline_options = VlmPipelineOptions(vlm_options=vlm_options, enable_remote_services=True)
converter = DocumentConverter(
allowed_formats=[InputFormat.PDF],
format_options={
InputFormat.PDF: PdfFormatOption(
pipeline_cls=VlmPipeline, pipeline_options=pipeline_options
)
},
)
result = converter.convert(source=pdf_path, raises_on_error=True)
print("status: ", result.status)
print("errors: ", result.errors)
for p in result.pages:
vr = p.predictions.vlm_response
print(f"page {p.page_no}: text={vr.text!r} stop_reason={vr.stop_reason}")
print("markdown:", repr(result.document.export_to_markdown()))
```
Observed output (every API call failed, yet no exception is raised):
```
ERROR docling.utils.api_image_request: Error, could not process request: HTTPConnectionPool(host='127.0.0.1', port=9): Max retries exceeded with url: /v1/chat/completions (...)
status: ConversionStatus.SUCCESS
errors: []
page 1: text='' stop_reason=VlmStopReason.UNSPECIFIED
markdown: ''
```
The same happens with a reachable endpoint that read-times-out or returns a non-OpenAI error body (e.g. an Envoy `no healthy upstream` page) — those cases additionally never hit the #3515 retry (`read=0`) or exhaust it in ~3 s.
### Expected behavior
A page whose VLM call failed should be *detectable* from the `ConversionResult`, at minimum:
1. Return a distinct stop reason for swallowed API errors (e.g. `VlmStopReason.API_ERROR` rather than overloading `UNSPECIFIED`), and have `VlmPipeline._determine_status()` treat it like `LENGTH`/`CONTENT_FILTERED`: append an `ErrorItem` and downgrade to `PARTIAL_SUCCESS` (or `FAILURE` when *all* pages failed). This mirrors the approach already taken for content filtering in #3051.
2. Optionally, an opt-in strict mode on the API engine options (e.g. `fail_on_api_error: bool`) that raises instead of returning empty, for pipelines where silently degraded output is worse than a hard failure.
3. Make the #3515 retry policy configurable (include read timeouts, allow larger backoff), since the hardcoded `read=0` / `backoff_factor=0.1` cannot cover transient endpoint outages.
Contributor guide
Research direction
Start with the failure handling in docling/utils/api_image_request.py, then trace stop reasons and status handling in docling/pipeline/vlm_pipeline.py and docling/document_converter.py. Run the supplied one-page PDF reproduction against the unavailable endpoint. Done means failed VLM calls are represented in ConversionResult instead of producing SUCCESS with empty page content, with the chosen strict or partial-failure behavior covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, api
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100