lllyasviel / lllyasviel/FramePack
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 29.74 GiB
- Dominant language
- Python
- Stars
- 17.3k
- Forks
- 1.7k
- PR merge metrics
- No merged PRs in 30d
Description
I am using a RTX 2080 Ti with 8GB of VRAM, it is weird : the model is requesting ~30GB of VRAM why ?
```
Using python3.12 (3.12.4)
Spawning shell within /home/elapouya/.cache/pypoetry/virtualenvs/framepack-root-zWKDNCUe-py3.12
❯ emulate bash -c '. /home/elapouya/.cache/pypoetry/virtualenvs/framepack-root-zWKDNCUe-py3.12/bin/activate'
❯ python ./demo_gradio.py
Currently enabled native sdp backends: ['flash', 'math', 'mem_efficient', 'cudnn']
Xformers is not installed!
Flash Attn is not installed!
Sage Attn is not installed!
Namespace(share=False, server='0.0.0.0', port=None, inbrowser=False)
Free VRAM 8.022705078125 GB
High-VRAM Mode: False
Downloading shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 3703.58it/s]
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 6.75it/s]
Fetching 3 files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 7206.71it/s]
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 4.10it/s]
transformer.high_quality_fp32_output_for_inference = True
* Running on local URL: http://0.0.0.0:7860
*
...
File "/home/elapouya/work/work/tmp/framepack_root/FramePack/diffusers_helper/models/hunyuan_video_packed.py", line 122, in attn_varlen_func
x = torch.nn.functional.scaled_dot_product_attention(q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2)).transpose(1, 2)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 29.74 GiB. GPU 0 has a total capacity of 10.75 GiB of which 3.98 GiB is free. Including non-PyTorch memory, this process has 3.85 GiB memory in use. Of the allocated memory 3.16 GiB is allocated by PyTorch, and 511.02 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with demo_gradio.py and the attention call in diffusers_helper/models/hunyuan_video_packed.py at line 122, then reproduce the allocation on the reported RTX 2080 Ti setup. Compare the reported VRAM usage with the attention inputs and determine whether this is expected behavior or a regression; done means a documented cause and a project-specific fix or clear reproduction result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100