docling-project / docling-project/docling

granite-docling-258M output is not parsed as doctags response_format

Open
#2,868 2 comments 1 reaction 0 assignees View on GitHub
bug triage/close-fixed
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
3d 4h
Merged PRs (30d)
98

Description

### Bug
Deploying granite-docling-258M via vllm on external gpu cluster.
We deployed docling serve via docker run.
Sending requests to docling using vlm api pipeline we didn't manage to let it run using response_format = doctags.

### Steps to reproduce
In the gpu cluster I deployed granite as follows:
vllm serve ibm-granite/granite-docling-258M --revision untied

Locally I'm running docling-serve-cpu image as follows:
`
docker run -p 5001:5001 \
-e DOCLING_SERVE_ENABLE_UI=true \
-e DOCLING_SERVE_ENABLE_REMOTE_SERVICES=true \
-e DOCLING_SERVE_MAX_SYNC_WAIT=1200 \
docling-serve-cpu-with-models:0.2.0 \
docling-serve -vv run
`

Running the following request:
`
curl -X 'POST' \
'http://127.0.0.1:5001/v1/convert/file' \
-H 'accept: application/json' \
-H 'Content-Type: multipart/form-data' \
-F 'pipeline=vlm' \
-F 'files=@DocLayNet-Dataset.pdf;type=application/pdf' \
-F 'from_formats=pdf' \
-F 'to_formats=md' \
-F 'to_formats=text' \
-F 'to_formats=doctags' \
-F 'image_export_mode=placeholder' \
-F 'abort_on_error=false' \
-F 'vlm_pipeline_model_api={
"concurrency": 4,
"headers": {
"Authorization": "Bearer xxx",
},
"params": {
"model": "ibm-granite/granite-docling-258M"
},
"prompt": "Convert this page to docling.",
"response_format": "doctags",
"scale": 2,
"temperature": 0,
"timeout": 1200,
"url": "https://DOMAIN_GPU_CLUSTER/v1/chat/completions"
}'
`

We made sure to able to reach the url. Granite model responses are looking good.

The output of this request is empty:
`
"document": {
"filename": "DocLayNet-Dataset.pdf",
"md_content": "",
"json_content": null,
"html_content": null,
"text_content": "",
"doctags_content": "\n"
},
"status": "success",
"errors": [],
"processing_time": 23.03597659499792,
"timings": {}
}
`

Instead testing response_format as markdown :
`
"document": {
"filename": "DocLayNet-Dataset.pdf",
"md_content": "<loc\\_14><loc\\_138><loc\\_30><loc\\_341>arXiv:2206.01062v1 [cs.CV] 2 Jun 2022 <loc\\_ ....
"json_content": null,
"html_content": null,
"text_content": "<loc\\_14><loc\\_138><loc\\
"doctags_content": "

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.