OpenImagingLab / OpenImagingLab/FlashVSR
long video error
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 152
- PR merge metrics
- No merged PRs in 30d
Description
- 3min input video
- infer_flashvsr_v1.1_tiny_long_video.py,scale 2x
(flashvsr) wzp@bms-b7bb00cb28e64685:/cx8k/fs101/wzp/code/0929/FlashVSR/examples/WanVSR$ python infer_flashvsr_v1.1_tiny_long_video.py
0 NVIDIA H800
Loading models from: ./FlashVSR-v1.1/diffusion_pytorch_model_streaming_dmd.safetensors
model_name: wan_video_dit model_class: WanModel
This model is initialized with extra kwargs: {'has_image_input': False, 'patch_size': [1, 2, 2], 'in_dim': 16, 'dim': 1536, 'ffn_dim': 8960, 'freq_dim': 256, 'text_dim': 4096, 'out_dim': 16, 'num_heads': 12, 'num_layers': 30, 'eps': 1e-06}
The following models are loaded: ['wan_video_dit'].
███████╗██╗ █████╗ ███████╗██╗ ██╗██╗ ██╗███████╗█████╗
██╔════╝██║ ██╔══██╗██╔════╝██║ ██║██║ ██║██╔════╝██╔══██╗
█████╗ ██║ ███████║███████╗███████║╚██╗ ██╔╝███████╗███████║
██╔══╝ ██║ ██╔══██║╚════██║██╔══██║ ╚████╔╝ ╚════██║██╔═██║
██║ ███████╗██║ ██║███████║██║ ██║ ╚██╔╝ ███████║██║ ██║
╚═╝ ╚══════╝╚═╝ ╚═╝╚══════╝╚═╝ ╚═╝ ╚═╝ ╚══════╝╚═╝ ╚═╝
⚡FlashVSR
Using wan_video_dit from ./FlashVSR-v1.1/diffusion_pytorch_model_streaming_dmd.safetensors.
No wan_video_vae models available.
<All keys matched successfully>
[frame-stitched-video_768p.mp4] Original Resolution: 1344x768 | Original Frames: 4267 | FPS: 24
[frame-stitched-video_768p.mp4] Scaled (x2.00): 2688x1536 -> Target (128-multiple): 2688x1536
[frame-stitched-video_768p.mp4] Target Frames (8n-3): 4261
/cx8k/fs101/wzp/code/0929/FlashVSR/examples/WanVSR/infer_flashvsr_v1.1_tiny_long_video.py:38: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /pytorch/torch/csrc/utils/tensor_numpy.cpp:203.)
t = torch.from_numpy(np.asarray(img, np.uint8)).to(device=device, dtype=torch.float32) # HWC
96%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 510/531 [11:45<00:29, 1.38s/it]
Traceback (most recent call last):
File "/cx8k/fs101/wzp/code/0929/FlashVSR/examples/WanVSR/infer_flashvsr_v1.1_tiny_long_video.py", line 238, in <module>
main()
File "/cx8k/fs101/wzp/code/0929/FlashVSR/examples/WanVSR/infer_flashvsr_v1.1_tiny_long_video.py", line 223, in main
video = pipe(
^^^^^
File "/cx8k/fs101/wzp/miniforge3/envs/flashvsr/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/cx8k/fs101/wzp/code/0929/FlashVSR/diffsynth/pipelines/flashvsr_tiny_long.py", line 389, in __call__
noise_pred_posi, pre_cache_k, pre_cache_v = model_fn_wan_video(
^^^^^^^^^^^^^^^^^^^
File "/cx8k/fs101/wzp/code/0929/FlashVSR/diffsynth/pipelines/flashvsr_tiny_long.py", line 554, in model_fn_wan_video
x, last_pre_cache_k, last_pre_cache_v = block(
^^^^^^
File "/cx8k/fs101/wzp/miniforge3/envs/flashvsr/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/cx8k/fs101/wzp/miniforge3/envs/flashvsr/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/cx8k/fs101/wzp/code/0929/FlashVSR/diffsynth/models/wan_video_dit.py", line 464, in forward
self_attn_output, self_attn_cache_k, self_attn_cache_v = self.self_attn(
^^^^^^^^^^^^^^^
File "/cx8k/fs101/wzp/miniforge3/envs/flashvsr/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/cx8k/fs101/wzp/miniforge3/envs/flashvsr/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/cx8k/fs101/wzp/code/0929/FlashVSR/diffsynth/models/wan_video_dit.py", line 330, in forward
q = rope_apply(q, freqs, self.num_heads)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/cx8k/fs101/wzp/code/0929/FlashVSR/diffsynth/models/wan_video_dit.py", line 269, in rope_apply
x_out = torch.view_as_real(x_out * freqs).flatten(2)
~~~~~~^~~~~~~
RuntimeError: The size of tensor a (64) must match the size of tensor b (42) at non-singleton dimension 3
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure with examples/WanVSR/infer_flashvsr_v1.1_tiny_long_video.py using the reported 3-minute, 2x input. Start at diffsynth/models/wan_video_dit.py:269 in rope_apply and trace the tensor shapes from the call at line 330, then inspect diffsynth/pipelines/flashvsr_tiny_long.py around line 389. Done means the example processes this input without the dimension-mismatch error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- computer-vision
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100