docling-project / docling-project/docling

VLM API call failures are silently converted to empty pages: ConversionStatus.SUCCESS, empty errors, and raises_on_error=True never fires

Open
#4,009 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

When the remote VLM endpoint fails during page conversion in the VLM pipeline (read timeout, connection error, or a gateway error body that fails response parsing), the failure is swallowed and the affected pages come back **empty**, while the conversion reports **complete success**. There is no programmatic way to distinguish "this document is genuinely blank" from "every API call failed".

The swallow chain:

1. `api_image_request()` catches **every** exception, logs it, and returns empty text as a normal result ([api_image_request.py#L257-L259](https://github.com/docling-project/docling/blob/61d76f1ff3f8/docling/utils/api_image_request.py#L257-L259)):

```python
except Exception as e:
_log.error(f"Error, could not process request: {e}")
return ApiImageRequestResult("", 0, VlmStopReason.UNSPECIFIED)
```

2. The API inference engine wraps that into a normal `VlmEngineOutput`/`VlmPrediction` without checking text or stop reason.

3. `VlmPipeline._determine_status()` flags only `LENGTH` and `CONTENT_FILTERED` stop reasons (added in #3051) — `UNSPECIFIED`, the exact marker a swallowed exception produces, is not checked, so the status stays `SUCCESS` and `conv_res.errors` stays empty ([vlm_pipeline.py#L236-L268](https://github.com/docling-project/docling/blob/61d76f1ff3f8/docling/pipeline/vlm_pipeline.py#L236-L268)).

4. Even a downgraded status would not raise: `raises_on_error` only raises for statuses outside `{SUCCESS, PARTIAL_SUCCESS}` ([document_converter.py#L569-L581](https://github.com/docling-project/docling/blob/61d76f1ff3f8/docling/document_converter.py#L569-L581)).

The retry added in #3515 does not close this: `Retry(..., read=0, ...)` explicitly excludes read timeouts ([api_image_request.py#L29-L46](https://github.com/docling-project/docling/blob/61d76f1ff3f8/docling/utils/api_image_request.py#L29-L46)), the `backoff_factor=0.1` retries span only ~3 seconds (no help against an outage lasting minutes), and after exhaustion the same `except Exception` swallow applies.

The same transport-level swallow also affects `PictureDescriptionVlmEngineModel` (images silently get `""` descriptions), but the page-conversion case is the destructive one: the empty text *is* the page content.

**Real-world impact:** we run docling as the conversion step of a document-ingestion service feeding a RAG index. During a temporary outage of the gateway in front of our VLM endpoint, every per-page call hit the 180 s read timeout; multi-page PDFs "converted successfully" with every page blank, and the pipeline committed empty documents into the index — overwriting previously good content, since our flow deletes-then-reingests. Roughly a hundred documents were silently emptied in a 15-minute window, discovered only days later when retrieval returned nothing for them. Nothing in `ConversionResult` (status, errors, document) distinguished them from genuinely empty files; the only trace was ERROR lines on the `docling.utils.api_image_request` logger, which we now scrape with a `logging.Handler` as a workaround.

### Steps to reproduce

```python
from pypdf import PdfWriter

pdf_path = "/tmp/one_page.pdf"
w = PdfWriter()
w.add_blank_page(width=200, height=200)
with open(pdf_path, "wb") as f:
w.write(f)

from docling.datamodel.base_models import InputFormat
from docling.datamodel.pipeline_options import VlmConvertOptions, VlmPipelineOptions
from docling.datamodel.vlm_engine_options import ApiVlmEngineOptions, VlmEngineType
from docling.document_converter import DocumentConverter, PdfFormatOption
from docling.pipeline.vlm_pipeline import VlmPipeline

engine_options = ApiVlmEngineOptions(
runtime_type=VlmEngineType.API,
url="http://127.0.0.1:9/v1/chat/completions", # nothing listens here
concurrency=1,
timeout=5,
)
vlm_options = VlmConvertOptions.from_preset("granite_docling", engine_options=engine_options)
pipeline_options = VlmPipelineOptions(vlm_options=vlm_options, enable_remote_services=True)

converter = DocumentConverter(
allowed_formats=[InputFormat.PDF],
format_options={
InputFormat.PDF: PdfFormatOption(
pipeline_cls=VlmPipeline, pipeline_options=pipeline_options
)
},
)

result = converter.convert(source=pdf_path, raises_on_error=True)
print("status: ", result.status)
print("errors: ", result.errors)
for p in result.pages:
vr = p.predictions.vlm_response
print(f"page {p.page_no}: text={vr.text!r} stop_reason={vr.stop_reason}")
print("markdown:", repr(result.document.export_to_markdown()))
```

Observed output (every API call failed, yet no exception is raised):

```
ERROR docling.utils.api_image_request: Error, could not process request: HTTPConnectionPool(host='127.0.0.1', port=9): Max retries exceeded with url: /v1/chat/completions (...)
status: ConversionStatus.SUCCESS
errors: []
page 1: text='' stop_reason=VlmStopReason.UNSPECIFIED
markdown: ''
```

The same happens with a reachable endpoint that read-times-out or returns a non-OpenAI error body (e.g. an Envoy `no healthy upstream` page) — those cases additionally never hit the #3515 retry (`read=0`) or exhaust it in ~3 s.

### Expected behavior

A page whose VLM call failed should be *detectable* from the `ConversionResult`, at minimum:

1. Return a distinct stop reason for swallowed API errors (e.g. `VlmStopReason.API_ERROR` rather than overloading `UNSPECIFIED`), and have `VlmPipeline._determine_status()` treat it like `LENGTH`/`CONTENT_FILTERED`: append an `ErrorItem` and downgrade to `PARTIAL_SUCCESS` (or `FAILURE` when *all* pages failed). This mirrors the approach already taken for content filtering in #3051.
2. Optionally, an opt-in strict mode on the API engine options (e.g. `fail_on_api_error: bool`) that raises instead of returning empty, for pipelines where silently degraded output is worse than a hard failure.
3. Make the #3515 retry policy configurable (include read timeouts, allow larger backoff), since the hardcoded `read=0` / `backoff_factor=0.1` cannot cover transient endpoint outages.

Contributor guide

Open the contributing guide

Research direction

Start with the failure handling in docling/utils/api_image_request.py, then trace stop reasons and status handling in docling/pipeline/vlm_pipeline.py and docling/document_converter.py. Run the supplied one-page PDF reproduction against the unavailable endpoint. Done means failed VLM calls are represented in ConversionResult instead of producing SUCCESS with empty page content, with the chosen strict or partial-failure behavior covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, api
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.