spring-projects / spring-projects/spring-ai

Can someone help When I upload an MP4 video, there will be an error that recognizes the video as an image

Open
#4,577 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

status: waiting-for-triage
Dominant language
Java
Stars
9.5k
Forks
2.9k
Avg merge
1d 7h
Merged PRs (30d)
6

Description

LLM:Qwen2.5-Omni-7B-AWQ
spring-ai: 1.1.0-M2

My core code:

`String mime;
String suffix = requestPojo.getFileName().split("\.")[1];
switch (suffix) {
case "png":
mime = "image/png";
break;
case "mp4":
mime = "video/mp4";
break;
default:
mime = "image/jpeg";
}

        String finalMime = mime;
        promptUserSpec = u -> u
                .text(prompt1.getContents())
                .media(
                        Media.builder()
                                .mimeType(MimeType.valueOf(finalMime))
                                .data(new FileSystemResource(uploadPath + File.separator +requestPojo.getFileName()))
                                .build()
        );`

After uploading to the large model, the error message is as follows:

(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] Error in preprocessing prompt inputs
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] Traceback (most recent call last):
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] File "/home/venv_vllm_new/lib/python3.12/site-packages/vllm/multimodal/utils.py", line 244, in fetch_image_async
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] return await self.load_from_url_async(
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] File "/home/venv_vllm_new/lib/python3.12/site-packages/vllm/multimodal/utils.py", line 165, in load_from_url_async
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] return await future
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] ^^^^^^^^^^^^
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] result = self.fn(*self.args, **self.kwargs)
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] File "/home/venv_vllm_new/lib/python3.12/site-packages/vllm/multimodal/utils.py", line 100, in _load_data_url
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] return media_io.load_base64(media_type, data)
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] File "/home/venv_vllm_new/lib/python3.12/site-packages/vllm/multimodal/image.py", line 91, in load_base64
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] return self.load_bytes(pybase64.b64decode(data, validate=True))
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] File "/home/venv_vllm_new/lib/python3.12/site-packages/vllm/multimodal/image.py", line 86, in load_bytes
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] image = Image.open(BytesIO(data))
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] ^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] File "/home/venv_vllm_new/lib/python3.12/site-packages/PIL/Image.py", line 3572, in open
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] raise UnidentifiedImageError(msg)
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] PIL.UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x78fdf9a06610>
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245]
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] The above exception was the direct cause of the following exception:
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245]
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] Traceback (most recent call last):
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] File "/home/venv_vllm_new/lib/python3.12/site-packages/vllm/entrypoints/openai/serving_chat.py", line 220, in create_chat_completion
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] ) = await self._preprocess_chat(
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] File "/home/venv_vllm_new/lib/python3.12/site-packages/vllm/entrypoints/openai/serving_engine.py", line 903, in _preprocess_chat
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] mm_data = await mm_data_future
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] ^^^^^^^^^^^^^^^^^^^^
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] File "/home/venv_vllm_new/lib/python3.12/site-packages/vllm/entrypoints/chat_utils.py", line 612, in all_mm_data
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] modality: await asyncio.gather(*items)
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] File "/home/venv_vllm_new/lib/python3.12/site-packages/vllm/multimodal/utils.py", line 251, in fetch_image_async
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] raise ValueError(str(e)) from e
(APIServer pid=196656) ERROR 10-09 16:17:15 [serving_chat.py:245] ValueError: cannot identify image file <_io.BytesIO object at 0x78fdf9a06610>

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reported Spring AI code using Media.builder().mimeType() and compare it with the vLLM traceback through multimodal/utils.py and image.py. Determine whether video/mp4 is supported by this integration and identify the repository behavior or documentation that should change. Done means the MP4 handling is either supported and tested or clearly documented as unsupported.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spring
Domain
ai, backend-api-design
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.