huggingface / huggingface/diffusers

High memory consumption for HunyuanVideo on CPU

Aperta
#10,569 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub
bug stale
Lingua principale
Python
Stelle
34.5k
Fork
7.3k
Merge medio
3g 3h
PR unite (30g)
91

Descrizione

### Describe the bug

We have a 4th-gen Xeon Scalable system that we are trying to run HunyuanVideo (via Diffusers) on. Remarkably, the demo code runs out of the box with no tweaks, which is a testament to the quality of the Intel PyTorch code :) However, during inference we see very high memory usage - over 180GB - to generate a 512x320x129 frame video; this is 3x what the CUDA implementation needs to generate a 1280x720x129 frame video. My suspicion is that this is because PyTorch SDPA doesn't have a memory efficient implementation on CPU.

### Reproduction

```
import torch
from diffusers import HunyuanVideoPipeline, HunyuanVideoTransformer3DModel
from diffusers.utils import export_to_video

model_id = "hunyuanvideo-community/HunyuanVideo"
transformer = HunyuanVideoTransformer3DModel.from_pretrained(
model_id, subfolder="transformer", torch_dtype=torch.bfloat16
)

pipe = HunyuanVideoPipeline.from_pretrained(model_id, transformer=transformer, torch_dtype=torch.float16)
pipe.vae.enable_tiling()

with torch.no_grad():
output = pipe(
prompt="A walk through tokyo at night, realistic",
height=320,
width=512,
num_frames=129,
num_inference_steps=50,
).frames[0]
export_to_video(output, "output4.mp4", fps=24)
```

### System Info

- 🤗 Diffusers version: 0.33.0.dev0
- Platform: Linux-6.5.0-41-generic-x86_64-with-glibc2.35
- Running on Google Colab?: No
- Python version: 3.10.12
- PyTorch version (GPU?): 2.6.0.dev20241225+cpu (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Huggingface_hub version: 0.27.0
- Transformers version: 4.46.3
- Accelerate version: 1.2.1
- PEFT version: 0.6.0
- Bitsandbytes version: not installed
- Safetensors version: 0.4.5
- xFormers version: not installed
- Accelerator: NA
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@DN6 @a-r-r-o-w

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Start by running the provided HunyuanVideoPipeline reproduction on the stated CPU-only PyTorch and Diffusers versions, then investigate the suspected CPU SDPA memory behavior. Done means explaining the excessive memory use and identifying a validated way to reduce it while preserving video generation.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python, pytorch
Ambito
machine-learning, performance
Tipo di issue
Bug
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
25/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.