docling-project / docling-project/docling

Incorrect artifact directory parsing, leading to `ValueError: Unrecognized model` when `smolvlm` is required

Open
#2,987 4 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

### Bug
After downloading docling models to the artifacts path, the document parsing fails if the pipeline requires `smolvlm`, even after manually downloading it.

### Steps to reproduce
Use this code to specify the pipeline, name it `test.py`:
```py
from docling.datamodel.base_models import InputFormat
from docling.datamodel.pipeline_options import PdfPipelineOptions
from docling.document_converter import DocumentConverter, PdfFormatOption

ARTIFACTS_PATH = "./models"

PIPELINE_OPTIONS_STANDARD = PdfPipelineOptions(
artifacts_path = ARTIFACTS_PATH,
do_picture_classification = True,
do_picture_description = True,
do_chart_extraction = True,
generate_picture_images = True,
do_table_structure = True,
do_ocr = True,
do_code_enrichment = False,
do_formula_enrichment = False,
generate_table_images = False,
ocr_batch_size = 4,
layout_batch_size = 4,
table_batch_size = 4,
batch_polling_interval_seconds = 0.5,
queue_max_size = 100
)

class PDFProcessor:
def __init__(self):
self.converter = DocumentConverter(
format_options={
InputFormat.PDF: PdfFormatOption(
pipeline_options = PIPELINE_OPTIONS_STANDARD
)
}
)

def process_pdf(self, pdf_path: str) -> str:
document = self.converter.convert(pdf_path).document
return document.export_to_markdown()

document_pdf = "any.pdf"

parser = PDFProcessor()
markdown = parser.process_pdf(document_pdf)

with open("output.md", "w") as file:
file.write(markdown)
```

Now download docling base models with:
```sh
docling-tools models download --output-dir "./models"
```

Then try to run the `test.py` specified above, you will get:
```
FileNotFoundError: Model 'HuggingFaceTB/SmolVLM-256M-Instruct' not found in artifacts_path.
Expected location: models/HuggingFaceTB--SmolVLM-256M-Instruct
Available models in models/: docling-project--DocumentFigureClassifier-v2.0, docling-project--CodeFormulaV2, docling-project--docling-layout-heron, RapidOcr, docling-project--docling-models

To fix this issue:
1. Download the model: docling-tools models download-hf-repo HuggingFaceTB/SmolVLM-256M-Instruct
2. Or remove --artifacts-path to enable auto-download
3. Or use a different model that exists in your artifacts_path
```

This is not an issue, we can dowload `smolvlm` with the following:
```sh
docling-tools models download --output-dir "./models" smolvlm
```

But then, if we run `test.py` again, we get:
```
ValueError: Unrecognized model in models. Should have a `model_type` key in its config.json, or contain one of the following strings in its name: aimv2, aimv2_vision_model, albert, align, altclip, [...], zamba, zamba2, zoedepth
```

However this is not the case; `HuggingFaceTB--SmolVLM-256M-Instruct` has a `config.json` file and the json contains the `model_type` key:
```json
{
"architectures": [
"Idefics3ForConditionalGeneration"
],
"image_token_id": 49190,
"model_type": "idefics3",
"scale_factor": 4,
[...]
}
```

### Docling version
```
Docling version: 2.73.0
Docling Core version: 2.64.0
Docling IBM Models version: 3.11.0
Docling Parse version: 4.7.3
Python: cpython-312 (3.12.12)
Platform: macOS-15.7.1-arm64-arm-64bit
```

### Python version
```
Python 3.12.12
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.