huggingface / huggingface/diffusers

[Pipeline] AnimateDiff + SparseControl + ControlNet

Aperta
#9,329 17 commenti 0 reazioni 0 assegnatari Vedi su GitHub
community-examples consider-for-modular-diffusers contributions-welcome good first issue
Lingua principale
Python
Stelle
34.5k
Fork
7.3k
Merge medio
3g 3h
PR unite (30g)
91

Descrizione

As the title suggests, we would like a pipeline that supports all three techniques. Currently, we have standalone pipelines for [SparseCtrl](https://github.com/huggingface/diffusers/blob/e417d028115e72b953a73e39d9687aa70ba3e37e/src/diffusers/pipelines/animatediff/pipeline_animatediff_sparsectrl.py) and [ControlNet](https://github.com/huggingface/diffusers/blob/e417d028115e72b953a73e39d9687aa70ba3e37e/src/diffusers/pipelines/animatediff/pipeline_animatediff_controlnet.py). A combination of the two might be interesting to see!

Right now video prediction is hard to control as the new frames are highly dependent on the prompt, if we could use images we would have better/finer control. This pipeline would enable apps like Blender to generate new images based on past reference frames and a depth buffer.

Looking at the code it looks like this is doable but before I try, I would like to get the input and suggestions of more expert people on this possible approach (@a-r-r-o-w or @DN6 :) ):

1. make [pipeline_animatediff_sparsectrl.py](https://github.com/huggingface/diffusers/blob/e417d028115e72b953a73e39d9687aa70ba3e37e/src/diffusers/pipelines/animatediff/pipeline_animatediff_sparsectrl.py) and [pipeline_animatediff_controlnet.py](https://github.com/huggingface/diffusers/blob/e417d028115e72b953a73e39d9687aa70ba3e37e/src/diffusers/pipelines/animatediff/pipeline_animatediff_controlnet.py) as similar as possible so diffing shows as much as common code as possible
2. refactor the blocks of code that are different into functions
3. have these functions work together in a new single pipeline

Does this make sense?

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Start by comparing src/diffusers/pipelines/animatediff/pipeline_animatediff_sparsectrl.py with pipeline_animatediff_controlnet.py, focusing on their shared and differing pipeline logic. Review the open discussion for expert guidance before deciding how to combine them; done means a single pipeline supports AnimateDiff, SparseControl, and ControlNet together.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python, pytorch
Ambito
machine-learning
Tipo di issue
Funzionalità
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Attiva
Chiarezza
Abbastanza chiara
Idoneità per principianti
45/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.