docling-project / docling-project/docling

feat: share VLM model instance across pipeline stages when repo_id and dtype match

Open
#3,524 1 comment 0 reactions 0 assignees View on GitHub
enhancement vlm-pipeline
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

### Requested feature
When `do_picture_description=True` and `do_chart_extraction=True` are both enabled in the same pipeline, and both stages are configured to use the same underlying model (e.g. `ibm-granite/granite-vision-4.1-4b`), docling loads two independent model instances into GPU memory.

Both `ChartExtractionModelGraniteVisionV4._load_model()` and `PictureDescriptionVlmModel.__init__()` call `AutoModelForImageTextToText.from_pretrained()` independently. There is no shared model registry or cache between pipeline stages. The pipeline assembly in `base_pipeline.py` instantiates each stage separately with no deduplication check.

On constrained hardware, running both stages with the same 4B model pushes static VRAM to ~18 GB, leaving very little headroom for activations during inference. A shared model registry keyed by (repo_id, revision, dtype, device) within a single pipeline instance would allow both stages to reuse the same in-memory weights, cutting the combined cost roughly in half.

### Alternatives
- Use different models per stage (e.g. `granite-vision-3.3-2b` for description, `granite-vision-4.1-4b` for chart extraction) avoids the duplication but forces a quality trade-off that shouldn't be necessary
- Disable one of the two stages, but loses either picture descriptions or structured chart data

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.