aigc-apps / aigc-apps/VideoX-Fun

使用 CogVideoX-Fun-V1.1-5b-control 推理时失败

Open
#120 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.3k
Forks
188
Avg merge
2d 22h
Merged PRs (30d)
3

Description

你好,当我使用CogVideoX-Fun-V1.1-5b-control 模型进行 predict_v2v_control.py 推理时候报错:
RuntimeError: mat1 and mat2 shapes cannot be multiplied (3456x256 and 264x3072)
报错发生在 cogvideox/models/transformer3d.py", line 147 image_embeds = self.proj(image_embeds) 这句
请问这个是代码不支持V1.1-5b-control模型吗

具体报错信息如下:
Traceback (most recent call last):
File "/seaweedfs/training/experiments/zaiyou/envs/torch2.5.1/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/seaweedfs/training/experiments/zaiyou/envs/torch2.5.1/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/root/.vscode-server/extensions/ms-python.debugpy-2025.0.0/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/main.py", line 71, in
cli.main()
File "/root/.vscode-server/extensions/ms-python.debugpy-2025.0.0/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 501, in main
run()
File "/root/.vscode-server/extensions/ms-python.debugpy-2025.0.0/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 351, in run_file
runpy.run_path(target, run_name="main")
File "/root/.vscode-server/extensions/ms-python.debugpy-2025.0.0/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 310, in run_path
return _run_module_code(code, init_globals, run_name, pkg_name=pkg_name, script_name=fname)
File "/root/.vscode-server/extensions/ms-python.debugpy-2025.0.0/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 127, in _run_module_code
_run_code(code, mod_globals, init_globals, mod_name, mod_spec, pkg_name, script_name)
File "/root/.vscode-server/extensions/ms-python.debugpy-2025.0.0/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 118, in _run_code
exec(code, run_globals)
File "/seaweedfs/training/experiments/zaiyou/eeds_code/ComfyUI/custom_nodes/CogVideoX-Fun/predict_v2v_control.py", line 138, in
sample = pipeline(
File "/seaweedfs/training/experiments/zaiyou/envs/torch2.5.1/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/seaweedfs/training/experiments/zaiyou/eeds_code/ComfyUI/custom_nodes/CogVideoX-Fun/cogvideox/pipeline/pipeline_cogvideox_control.py", line 906, in call
noise_pred = self.transformer(
File "/seaweedfs/training/experiments/zaiyou/envs/torch2.5.1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/seaweedfs/training/experiments/zaiyou/envs/torch2.5.1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/seaweedfs/training/experiments/zaiyou/envs/torch2.5.1/lib/python3.10/site-packages/accelerate/hooks.py", line 170, in new_forward
output = module._old_forward(*args, **kwargs)
File "/seaweedfs/training/experiments/zaiyou/eeds_code/ComfyUI/custom_nodes/CogVideoX-Fun/cogvideox/models/transformer3d.py", line 589, in forward
hidden_states = self.patch_embed(encoder_hidden_states, hidden_states)
File "/seaweedfs/training/experiments/zaiyou/envs/torch2.5.1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/seaweedfs/training/experiments/zaiyou/envs/torch2.5.1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/seaweedfs/training/experiments/zaiyou/eeds_code/ComfyUI/custom_nodes/CogVideoX-Fun/cogvideox/models/transformer3d.py", line 147, in forward
image_embeds = self.proj(image_embeds)
File "/seaweedfs/training/experiments/zaiyou/envs/torch2.5.1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/seaweedfs/training/experiments/zaiyou/envs/torch2.5.1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/seaweedfs/training/experiments/zaiyou/envs/torch2.5.1/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 125, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (3456x256 and 264x3072)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure through predict_v2v_control.py, then inspect cogvideox/models/transformer3d.py at line 147 and the control pipeline entry point in pipeline_cogvideox_control.py. Compare the image embedding dimensions from CogVideoX-Fun-V1.1-5b-control with the projection layer configuration. Done means establishing compatible inference or documenting that this model is unsupported with the current code.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.