OpenBMB / OpenBMB/MiniCPM-V

Minicpm_o-4-5进行视频推理和多模混合推理时报错

Open
#1,081 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
26.4k
Forks
2.1k
Avg merge
14h 39m
Merged PRs (30d)
3

Description

File "/workdir/user_repository/llm-eval-public/inference/local_deploy_minicpm_o_4_5.py", line 115, in generate
answer = self.model.chat(msgs=messages, max_new_tokens=32768,max_inp_length=32000,max_slice_nums=1, omni_mode=True, use_tts_template=False,
File "/usr/local/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/home/hadoop-aipnlp/.cache/huggingface/modules/transformers_modules/main/modeling_minicpmo.py", line 1202, in chat
res, outputs = self.generate(
File "/usr/local/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/home/hadoop-aipnlp/.cache/huggingface/modules/transformers_modules/main/modeling_minicpmo.py", line 892, in generate
model_inputs["inputs_embeds"] = self.get_omni_embedding(
File "/home/hadoop-aipnlp/.cache/huggingface/modules/transformers_modules/main/modeling_minicpmo.py", line 771, in get_omni_embedding
input_embeddings[i, bound[0] : bound[1]] = audio_embs[
RuntimeError: The expanded size of the tensor (21) must match the existing size (20) at non-singleton dimension 0. Target sizes: [21, 4096]. Tensor sizes: [20, 4096]

视频推理的报错:
File "/workdir/user_repository/llm-eval-public/inference/local_deploy_minicpm_o_4_5.py", line 115, in generate
answer = self.model.chat(msgs=messages, max_new_tokens=32768,max_inp_length=32000,max_slice_nums=1, omni_mode=True, use_tts_template=False,
File "/usr/local/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/home/hadoop-aipnlp/.cache/huggingface/modules/transformers_modules/main/modeling_minicpmo.py", line 1146, in chat
content = normalize_content(content)
File "/home/hadoop-aipnlp/.cache/huggingface/modules/transformers_modules/main/utils.py", line 2400, in normalize_content
normalized = normalize_content_item(item)
File "/home/hadoop-aipnlp/.cache/huggingface/modules/transformers_modules/main/utils.py", line 2338, in normalize_content_item
video_frames, audio_segments, stacked_frames = get_video_frame_audio_segments(
File "/home/hadoop-aipnlp/.local/lib/python3.10/site-packages/minicpmo/utils.py", line 392, in get_video_frame_audio_segments
video_segments, timestamps = _extract_frames_by_timestamps(timestamps, is_long_video)
File "/home/hadoop-aipnlp/.local/lib/python3.10/site-packages/minicpmo/utils.py", line 357, in _extract_frames_by_timestamps
video = _vr.get_batch(frame_idx).asnumpy()
File "/usr/local/conda/lib/python3.10/site-packages/decord/video_reader.py", line 175, in get_batch
arr = _CAPI_VideoReaderGetBatch(self._handle, indices)
File "/usr/local/conda/lib/python3.10/site-packages/decord/_ffi/_ctypes/function.py", line 173, in call
check_call(_LIB.DECORDFuncCall(
File "/usr/local/conda/lib/python3.10/site-packages/decord/_ffi/base.py", line 78, in check_call
raise DECORDError(err_str)
decord.ffi.base.DECORDError: [11:20:58] /github/workspace/src/video/ffmpeg/threaded_decoder.cc:104: Check failed: run.load()
请问下这两个有什么好的修复建议吗

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with inference/local_deploy_minicpm_o_4_5.py at line 115 and inspect the two failing paths in modeling_minicpmo.py, especially get_omni_embedding. For video, trace minicpmo/utils.py through get_video_frame_audio_segments and _extract_frames_by_timestamps into decord.get_batch. Done means both multimodal and video inference complete without either reported error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.