huggingface / huggingface/diffusers
High memory consumption for HunyuanVideo on CPU
- Lingua principale
- Python
- Stelle
- 34.5k
- Fork
- 7.3k
- Merge medio
- 3g 3h
- PR unite (30g)
- 91
Descrizione
### Describe the bug
We have a 4th-gen Xeon Scalable system that we are trying to run HunyuanVideo (via Diffusers) on. Remarkably, the demo code runs out of the box with no tweaks, which is a testament to the quality of the Intel PyTorch code :) However, during inference we see very high memory usage - over 180GB - to generate a 512x320x129 frame video; this is 3x what the CUDA implementation needs to generate a 1280x720x129 frame video. My suspicion is that this is because PyTorch SDPA doesn't have a memory efficient implementation on CPU.
### Reproduction
```
import torch
from diffusers import HunyuanVideoPipeline, HunyuanVideoTransformer3DModel
from diffusers.utils import export_to_video
model_id = "hunyuanvideo-community/HunyuanVideo"
transformer = HunyuanVideoTransformer3DModel.from_pretrained(
model_id, subfolder="transformer", torch_dtype=torch.bfloat16
)
pipe = HunyuanVideoPipeline.from_pretrained(model_id, transformer=transformer, torch_dtype=torch.float16)
pipe.vae.enable_tiling()
with torch.no_grad():
output = pipe(
prompt="A walk through tokyo at night, realistic",
height=320,
width=512,
num_frames=129,
num_inference_steps=50,
).frames[0]
export_to_video(output, "output4.mp4", fps=24)
```
### System Info
- 🤗 Diffusers version: 0.33.0.dev0
- Platform: Linux-6.5.0-41-generic-x86_64-with-glibc2.35
- Running on Google Colab?: No
- Python version: 3.10.12
- PyTorch version (GPU?): 2.6.0.dev20241225+cpu (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Huggingface_hub version: 0.27.0
- Transformers version: 4.46.3
- Accelerate version: 1.2.1
- PEFT version: 0.6.0
- Bitsandbytes version: not installed
- Safetensors version: 0.4.5
- xFormers version: not installed
- Accelerator: NA
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
### Who can help?
@DN6 @a-r-r-o-w
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Start by running the provided HunyuanVideoPipeline reproduction on the stated CPU-only PyTorch and Diffusers versions, then investigate the suspected CPU SDPA memory behavior. Done means explaining the excessive memory use and identifying a validated way to reduce it while preserving video generation.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- python, pytorch
- Ambito
- machine-learning, performance
- Tipo di issue
- Bug
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Stato di attività
- Ferma
- Chiarezza
- Da chiarire
- Idoneità per principianti
- 25/100