OpenImagingLab / OpenImagingLab/FlashVSR

A cuda error when run your model

Open
#10 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.9k
Forks
152
PR merge metrics
No merged PRs in 30d

Description

Traceback (most recent call last):
File "/root/autodl-tmp/code/FlashVSR/examples/WanVSR/infer_flashvsr_full.py", line 218, in
main()
File "/root/autodl-tmp/code/FlashVSR/examples/WanVSR/infer_flashvsr_full.py", line 204, in main
video = pipe(
^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/code/FlashVSR/diffsynth/pipelines/flashvsr_full.py", line 407, in call
noise_pred_posi, pre_cache_k, pre_cache_v = model_fn_wan_video(
^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/code/FlashVSR/diffsynth/pipelines/flashvsr_full.py", line 570, in model_fn_wan_video
x, last_pre_cache_k, last_pre_cache_v = block(
^^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/code/FlashVSR/diffsynth/models/wan_video_dit.py", line 461, in forward
self_attn_output, self_attn_cache_k, self_attn_cache_v = self.self_attn(
^^^^^^^^^^^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/code/FlashVSR/diffsynth/models/wan_video_dit.py", line 362, in forward
x = self.attn(reorder_q, reorder_k, reorder_v, attention_mask)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/code/FlashVSR/diffsynth/models/wan_video_dit.py", line 296, in forward
x = flash_attention(q=q, k=k, v=v, num_heads=self.num_heads, attention_mask=attention_mask)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/code/FlashVSR/diffsynth/models/wan_video_dit.py", line 189, in flash_attention
x = block_sparse_attn_func(
^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/block_sparse_attn/block_sparse_attn_interface.py", line 399, in block_sparse_attn_func
return func.apply(
^^^^^^^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/torch/autograd/function.py", line 575, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/block_sparse_attn/block_sparse_attn_interface.py", line 192, in forward
out, q, k, v, out_padded, softmax_lse, S_dmask, rng_state = _block_sparse_attn_forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/block_sparse_attn/block_sparse_attn_interface.py", line 98, in _block_sparse_attn_forward
out, q, k, v, out_padded, softmax_lse, S_dmask, rng_state = block_sparse_attn_cuda.fwd_block(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

CUDA Version: 12.4
GPU: 1 x H800
The envrionment is by your readme, can you tell me is there any miss for my env?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with examples/WanVSR/infer_flashvsr_full.py and diffsynth/models/wan_video_dit.py, then inspect the block_sparse_attn dependency used by flash_attention. Reproduce the failure on the reported H800 with CUDA 12.4 and compare the environment against the README. Done means identifying the incompatible or missing environment requirement and documenting the verified setup or required change.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.