OpenImagingLab / OpenImagingLab/FlashVSR

long video error

Open
#112 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.9k
Forks
152
PR merge metrics
No merged PRs in 30d

Description

  • 3min input video
  • infer_flashvsr_v1.1_tiny_long_video.py,scale 2x
(flashvsr) wzp@bms-b7bb00cb28e64685:/cx8k/fs101/wzp/code/0929/FlashVSR/examples/WanVSR$ python infer_flashvsr_v1.1_tiny_long_video.py
0 NVIDIA H800
Loading models from: ./FlashVSR-v1.1/diffusion_pytorch_model_streaming_dmd.safetensors
    model_name: wan_video_dit model_class: WanModel
        This model is initialized with extra kwargs: {'has_image_input': False, 'patch_size': [1, 2, 2], 'in_dim': 16, 'dim': 1536, 'ffn_dim': 8960, 'freq_dim': 256, 'text_dim': 4096, 'out_dim': 16, 'num_heads': 12, 'num_layers': 30, 'eps': 1e-06}
    The following models are loaded: ['wan_video_dit'].

███████╗██╗      █████╗ ███████╗██╗  ██╗██╗   ██╗███████╗█████╗
██╔════╝██║     ██╔══██╗██╔════╝██║  ██║██║   ██║██╔════╝██╔══██╗
█████╗  ██║     ███████║███████╗███████║╚██╗ ██╔╝███████╗███████║
██╔══╝  ██║     ██╔══██║╚════██║██╔══██║ ╚████╔╝ ╚════██║██╔═██║
██║     ███████╗██║  ██║███████║██║  ██║  ╚██╔╝  ███████║██║  ██║
╚═╝     ╚══════╝╚═╝  ╚═╝╚══════╝╚═╝  ╚═╝   ╚═╝   ╚══════╝╚═╝  ╚═╝
                         ⚡FlashVSR

Using wan_video_dit from ./FlashVSR-v1.1/diffusion_pytorch_model_streaming_dmd.safetensors.
No wan_video_vae models available.
<All keys matched successfully>
[frame-stitched-video_768p.mp4] Original Resolution: 1344x768 | Original Frames: 4267 | FPS: 24
[frame-stitched-video_768p.mp4] Scaled (x2.00): 2688x1536 -> Target (128-multiple): 2688x1536
[frame-stitched-video_768p.mp4] Target Frames (8n-3): 4261
/cx8k/fs101/wzp/code/0929/FlashVSR/examples/WanVSR/infer_flashvsr_v1.1_tiny_long_video.py:38: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /pytorch/torch/csrc/utils/tensor_numpy.cpp:203.)
  t = torch.from_numpy(np.asarray(img, np.uint8)).to(device=device, dtype=torch.float32)  # HWC
 96%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎        | 510/531 [11:45<00:29,  1.38s/it]
Traceback (most recent call last):
  File "/cx8k/fs101/wzp/code/0929/FlashVSR/examples/WanVSR/infer_flashvsr_v1.1_tiny_long_video.py", line 238, in <module>
    main()
  File "/cx8k/fs101/wzp/code/0929/FlashVSR/examples/WanVSR/infer_flashvsr_v1.1_tiny_long_video.py", line 223, in main
    video = pipe(
            ^^^^^
  File "/cx8k/fs101/wzp/miniforge3/envs/flashvsr/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/cx8k/fs101/wzp/code/0929/FlashVSR/diffsynth/pipelines/flashvsr_tiny_long.py", line 389, in __call__
    noise_pred_posi, pre_cache_k, pre_cache_v = model_fn_wan_video(
                                                ^^^^^^^^^^^^^^^^^^^
  File "/cx8k/fs101/wzp/code/0929/FlashVSR/diffsynth/pipelines/flashvsr_tiny_long.py", line 554, in model_fn_wan_video
    x, last_pre_cache_k, last_pre_cache_v = block(
                                            ^^^^^^
  File "/cx8k/fs101/wzp/miniforge3/envs/flashvsr/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/cx8k/fs101/wzp/miniforge3/envs/flashvsr/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/cx8k/fs101/wzp/code/0929/FlashVSR/diffsynth/models/wan_video_dit.py", line 464, in forward
    self_attn_output, self_attn_cache_k, self_attn_cache_v = self.self_attn(
                                                             ^^^^^^^^^^^^^^^
  File "/cx8k/fs101/wzp/miniforge3/envs/flashvsr/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/cx8k/fs101/wzp/miniforge3/envs/flashvsr/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/cx8k/fs101/wzp/code/0929/FlashVSR/diffsynth/models/wan_video_dit.py", line 330, in forward
    q = rope_apply(q, freqs, self.num_heads)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/cx8k/fs101/wzp/code/0929/FlashVSR/diffsynth/models/wan_video_dit.py", line 269, in rope_apply
    x_out = torch.view_as_real(x_out * freqs).flatten(2)
                               ~~~~~~^~~~~~~
RuntimeError: The size of tensor a (64) must match the size of tensor b (42) at non-singleton dimension 3
Image

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with examples/WanVSR/infer_flashvsr_v1.1_tiny_long_video.py using the reported 3-minute, 2x input. Start at diffsynth/models/wan_video_dit.py:269 in rope_apply and trace the tensor shapes from the call at line 330, then inspect diffsynth/pipelines/flashvsr_tiny_long.py around line 389. Done means the example processes this input without the dimension-mismatch error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
computer-vision
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.