docling-project / docling-project/docling
feat: share VLM model instance across pipeline stages when repo_id and dtype match
- Dominant language
- Python
- Stars
- 66.4k
- Forks
- 4.8k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 84
Description
### Requested feature
When `do_picture_description=True` and `do_chart_extraction=True` are both enabled in the same pipeline, and both stages are configured to use the same underlying model (e.g. `ibm-granite/granite-vision-4.1-4b`), docling loads two independent model instances into GPU memory.
Both `ChartExtractionModelGraniteVisionV4._load_model()` and `PictureDescriptionVlmModel.__init__()` call `AutoModelForImageTextToText.from_pretrained()` independently. There is no shared model registry or cache between pipeline stages. The pipeline assembly in `base_pipeline.py` instantiates each stage separately with no deduplication check.
On constrained hardware, running both stages with the same 4B model pushes static VRAM to ~18 GB, leaving very little headroom for activations during inference. A shared model registry keyed by (repo_id, revision, dtype, device) within a single pipeline instance would allow both stages to reuse the same in-memory weights, cutting the combined cost roughly in half.
### Alternatives
- Use different models per stage (e.g. `granite-vision-3.3-2b` for description, `granite-vision-4.1-4b` for chart extraction) avoids the duplication but forces a quality trade-off that shouldn't be necessary
- Disable one of the two stages, but loses either picture descriptions or structured chart data
Contributor guide
Assessment
This issue has not been assessed yet.