lllyasviel / lllyasviel/FramePack

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 29.74 GiB

Open
#419 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
17.3k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

I am using a RTX 2080 Ti with 8GB of VRAM, it is weird : the model is requesting ~30GB of VRAM why ?

```
Using python3.12 (3.12.4)
Spawning shell within /home/elapouya/.cache/pypoetry/virtualenvs/framepack-root-zWKDNCUe-py3.12
❯ emulate bash -c '. /home/elapouya/.cache/pypoetry/virtualenvs/framepack-root-zWKDNCUe-py3.12/bin/activate'
❯ python ./demo_gradio.py
Currently enabled native sdp backends: ['flash', 'math', 'mem_efficient', 'cudnn']
Xformers is not installed!
Flash Attn is not installed!
Sage Attn is not installed!
Namespace(share=False, server='0.0.0.0', port=None, inbrowser=False)
Free VRAM 8.022705078125 GB
High-VRAM Mode: False
Downloading shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 3703.58it/s]
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 6.75it/s]
Fetching 3 files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 7206.71it/s]
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 4.10it/s]
transformer.high_quality_fp32_output_for_inference = True
* Running on local URL: http://0.0.0.0:7860
*
...

File "/home/elapouya/work/work/tmp/framepack_root/FramePack/diffusers_helper/models/hunyuan_video_packed.py", line 122, in attn_varlen_func
x = torch.nn.functional.scaled_dot_product_attention(q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2)).transpose(1, 2)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 29.74 GiB. GPU 0 has a total capacity of 10.75 GiB of which 3.98 GiB is free. Including non-PyTorch memory, this process has 3.85 GiB memory in use. Of the allocated memory 3.16 GiB is allocated by PyTorch, and 511.02 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with demo_gradio.py and the attention call in diffusers_helper/models/hunyuan_video_packed.py at line 122, then reproduce the allocation on the reported RTX 2080 Ti setup. Compare the reported VRAM usage with the attention inputs and determine whether this is expected behavior or a regression; done means a documented cause and a project-specific fix or clear reproduction result.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.