OpenImagingLab / OpenImagingLab/FlashVSR
A cuda error when run your model
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 152
- PR merge metrics
- No merged PRs in 30d
Description
Traceback (most recent call last):
File "/root/autodl-tmp/code/FlashVSR/examples/WanVSR/infer_flashvsr_full.py", line 218, in
main()
File "/root/autodl-tmp/code/FlashVSR/examples/WanVSR/infer_flashvsr_full.py", line 204, in main
video = pipe(
^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/code/FlashVSR/diffsynth/pipelines/flashvsr_full.py", line 407, in call
noise_pred_posi, pre_cache_k, pre_cache_v = model_fn_wan_video(
^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/code/FlashVSR/diffsynth/pipelines/flashvsr_full.py", line 570, in model_fn_wan_video
x, last_pre_cache_k, last_pre_cache_v = block(
^^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/code/FlashVSR/diffsynth/models/wan_video_dit.py", line 461, in forward
self_attn_output, self_attn_cache_k, self_attn_cache_v = self.self_attn(
^^^^^^^^^^^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/code/FlashVSR/diffsynth/models/wan_video_dit.py", line 362, in forward
x = self.attn(reorder_q, reorder_k, reorder_v, attention_mask)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/code/FlashVSR/diffsynth/models/wan_video_dit.py", line 296, in forward
x = flash_attention(q=q, k=k, v=v, num_heads=self.num_heads, attention_mask=attention_mask)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/code/FlashVSR/diffsynth/models/wan_video_dit.py", line 189, in flash_attention
x = block_sparse_attn_func(
^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/block_sparse_attn/block_sparse_attn_interface.py", line 399, in block_sparse_attn_func
return func.apply(
^^^^^^^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/torch/autograd/function.py", line 575, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/block_sparse_attn/block_sparse_attn_interface.py", line 192, in forward
out, q, k, v, out_padded, softmax_lse, S_dmask, rng_state = _block_sparse_attn_forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/block_sparse_attn/block_sparse_attn_interface.py", line 98, in _block_sparse_attn_forward
out, q, k, v, out_padded, softmax_lse, S_dmask, rng_state = block_sparse_attn_cuda.fwd_block(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
CUDA Version: 12.4
GPU: 1 x H800
The envrionment is by your readme, can you tell me is there any miss for my env?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with examples/WanVSR/infer_flashvsr_full.py and diffsynth/models/wan_video_dit.py, then inspect the block_sparse_attn dependency used by flash_attention. Reproduce the failure on the reported H800 with CUDA 12.4 and compare the environment against the README. Done means identifying the incompatible or missing environment requirement and documenting the verified setup or required change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100