docling-project / docling-project/docling
granite-docling-258M output is not parsed as doctags response_format
- Dominant language
- Python
- Stars
- 66.4k
- Forks
- 4.8k
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 98
Description
### Bug
Deploying granite-docling-258M via vllm on external gpu cluster.
We deployed docling serve via docker run.
Sending requests to docling using vlm api pipeline we didn't manage to let it run using response_format = doctags.
### Steps to reproduce
In the gpu cluster I deployed granite as follows:
vllm serve ibm-granite/granite-docling-258M --revision untied
Locally I'm running docling-serve-cpu image as follows:
`
docker run -p 5001:5001 \
-e DOCLING_SERVE_ENABLE_UI=true \
-e DOCLING_SERVE_ENABLE_REMOTE_SERVICES=true \
-e DOCLING_SERVE_MAX_SYNC_WAIT=1200 \
docling-serve-cpu-with-models:0.2.0 \
docling-serve -vv run
`
Running the following request:
`
curl -X 'POST' \
'http://127.0.0.1:5001/v1/convert/file' \
-H 'accept: application/json' \
-H 'Content-Type: multipart/form-data' \
-F 'pipeline=vlm' \
-F 'files=@DocLayNet-Dataset.pdf;type=application/pdf' \
-F 'from_formats=pdf' \
-F 'to_formats=md' \
-F 'to_formats=text' \
-F 'to_formats=doctags' \
-F 'image_export_mode=placeholder' \
-F 'abort_on_error=false' \
-F 'vlm_pipeline_model_api={
"concurrency": 4,
"headers": {
"Authorization": "Bearer xxx",
},
"params": {
"model": "ibm-granite/granite-docling-258M"
},
"prompt": "Convert this page to docling.",
"response_format": "doctags",
"scale": 2,
"temperature": 0,
"timeout": 1200,
"url": "https://DOMAIN_GPU_CLUSTER/v1/chat/completions"
}'
`
We made sure to able to reach the url. Granite model responses are looking good.
The output of this request is empty:
`
"document": {
"filename": "DocLayNet-Dataset.pdf",
"md_content": "",
"json_content": null,
"html_content": null,
"text_content": "",
"doctags_content": "\n"
},
"status": "success",
"errors": [],
"processing_time": 23.03597659499792,
"timings": {}
}
`
Instead testing response_format as markdown :
`
"document": {
"filename": "DocLayNet-Dataset.pdf",
"md_content": "<loc\\_14><loc\\_138><loc\\_30><loc\\_341>arXiv:2206.01062v1 [cs.CV] 2 Jun 2022 <loc\\_ ....
"json_content": null,
"html_content": null,
"text_content": "<loc\\_14><loc\\_138><loc\\
"doctags_content": "
Contributor guide
Assessment
This issue has not been assessed yet.