huggingface / huggingface/diffusers

Wan 2.1 / Wan2.2 Lightning & Sparse Distillation & Tiny VAE Acceleration Methods

Aperta
#12,146 18 commenti 3 reazioni 0 assegnatari Vedi su GitHub
stale
Lingua principale
Python
Stelle
34.5k
Fork
7.3k
Merge medio
3g 3h
PR unite (30g)
91

Descrizione

Image

**Is your feature request related to a problem? Please describe.**
Some incredible acceleration methods have been released for both Wan2.1 and Wan2.2. These are currently appear to be unsupported with the official Diffusers pipelines.

- **[Wan2.2-Lightning](https://github.com/ModelTC/Wan2.2-Lightning)** enables 4-step inference for Wan2.2 via LoRAs
- **[LightX2V](https://github.com/ModelTC/lightx2v)** is related and also includes 4-step inference for Wan2.1 via LoRAs
- **[FastVideo](https://github.com/hao-ai-lab/FastVideo)** enables 3-step inference via distillation for Wan2.1 and Wan2.2 variants
- **[TAEHV](https://github.com/madebyollin/taehv?tab=readme-ov-file)** enables 4-6x increase in VAE decoding speed (2-3s to 0.5s)

**Describe the solution you'd like.**
1. The ability to load lightning LoRAs with Wan2.1 and Wan2.2 pipelines
2. The ability to load FastWan distilled models
3. The ability to load TAEHV as a Wan VAE alternative

**Describe alternatives you've considered.**

The [FastVideo](https://github.com/hao-ai-lab/FastVideo) and [LightX2V](https://github.com/ModelTC/lightx2v) official inference scripts.

TAEHV has a method for retro-fitting a [Diffusers pipeline](https://github.com/madebyollin/taehv/blob/main/examples/TAEW2.1_T2I_Demo.ipynb).

There are also various ComfyUI implementations.

**Additional context.**
Some of the LoRA acceleration might be supported already but is broken by recent changes to the LoRA loading ecosystem. Not certain. Also could use some guidance on editing the scheduler to allow for the time-step distillation.

I am currently investigating this space and figured I would open this issue to allow myself and others to post working examples and any relevant PRs. Thanks to all who do in advance.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Begin by comparing the FastVideo and LightX2V official inference scripts with the Wan2.1 and Wan2.2 Diffusers pipelines. Review the TAEHV TAEW2.1_T2I_Demo.ipynb retrofit example and investigate the reported LoRA-loading and scheduler questions. Done means the pipelines can load the requested Lightning LoRAs, FastWan distilled models, and TAEHV VAE alternative with working examples.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python, pytorch
Ambito
machine-learning, performance
Tipo di issue
Funzionalità
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
25/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.