spring-projects / spring-projects/spring-ai
Can someone help When I upload an MP4 video, there will be an error that recognizes the video as an image
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 9.5k
- Forks
- 2.9k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 6
Description
LLM:Qwen2.5-Omni-7B-AWQ
spring-ai: 1.1.0-M2
My core code:
`String mime;
String suffix = requestPojo.getFileName().split("\.")[1];
switch (suffix) {
case "png":
mime = "image/png";
break;
case "mp4":
mime = "video/mp4";
break;
default:
mime = "image/jpeg";
}
String finalMime = mime;
promptUserSpec = u -> u
.text(prompt1.getContents())
.media(
Media.builder()
.mimeType(MimeType.valueOf(finalMime))
.data(new FileSystemResource(uploadPath + File.separator +requestPojo.getFileName()))
.build()
);`
After uploading to the large model, the error message is as follows:
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] Error in preprocessing prompt inputs
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] Traceback (most recent call last):
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] File "/home/venv_vllm_new/lib/python3.12/site-packages/vllm/multimodal/utils.py", line 244, in fetch_image_async
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] return await self.load_from_url_async(
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] File "/home/venv_vllm_new/lib/python3.12/site-packages/vllm/multimodal/utils.py", line 165, in load_from_url_async
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] return await future
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] ^^^^^^^^^^^^
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] result = self.fn(*self.args, **self.kwargs)
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] File "/home/venv_vllm_new/lib/python3.12/site-packages/vllm/multimodal/utils.py", line 100, in _load_data_url
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] return media_io.load_base64(media_type, data)
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] File "/home/venv_vllm_new/lib/python3.12/site-packages/vllm/multimodal/image.py", line 91, in load_base64
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] return self.load_bytes(pybase64.b64decode(data, validate=True))
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] File "/home/venv_vllm_new/lib/python3.12/site-packages/vllm/multimodal/image.py", line 86, in load_bytes
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] image = Image.open(BytesIO(data))
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] ^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] File "/home/venv_vllm_new/lib/python3.12/site-packages/PIL/Image.py", line 3572, in open
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] raise UnidentifiedImageError(msg)
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] PIL.UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x78fdf9a06610>
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245]
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] The above exception was the direct cause of the following exception:
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245]
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] Traceback (most recent call last):
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] File "/home/venv_vllm_new/lib/python3.12/site-packages/vllm/entrypoints/openai/serving_chat.py", line 220, in create_chat_completion
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] ) = await self._preprocess_chat(
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] File "/home/venv_vllm_new/lib/python3.12/site-packages/vllm/entrypoints/openai/serving_engine.py", line 903, in _preprocess_chat
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] mm_data = await mm_data_future
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] ^^^^^^^^^^^^^^^^^^^^
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] File "/home/venv_vllm_new/lib/python3.12/site-packages/vllm/entrypoints/chat_utils.py", line 612, in all_mm_data
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] modality: await asyncio.gather(*items)
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] File "/home/venv_vllm_new/lib/python3.12/site-packages/vllm/multimodal/utils.py", line 251, in fetch_image_async
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] raise ValueError(str(e)) from e
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] ValueError: cannot identify image file <_io.BytesIO object at 0x78fdf9a06610>
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the reported Spring AI code using Media.builder().mimeType() and compare it with the vLLM traceback through multimodal/utils.py and image.py. Determine whether video/mp4 is supported by this integration and identify the repository behavior or documentation that should change. Done means the MP4 handling is either supported and tested or clearly documented as unsupported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spring
- Domain
- ai, backend-api-design
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100