huggingface / huggingface/diffusers

[Pipeline] AnimateDiff + SparseControl + ControlNet

Open
#9,329 17 comments 0 reactions 0 assignees View on GitHub
community-examples consider-for-modular-diffusers contributions-welcome good first issue
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

As the title suggests, we would like a pipeline that supports all three techniques. Currently, we have standalone pipelines for [SparseCtrl](https://github.com/huggingface/diffusers/blob/e417d028115e72b953a73e39d9687aa70ba3e37e/src/diffusers/pipelines/animatediff/pipeline_animatediff_sparsectrl.py) and [ControlNet](https://github.com/huggingface/diffusers/blob/e417d028115e72b953a73e39d9687aa70ba3e37e/src/diffusers/pipelines/animatediff/pipeline_animatediff_controlnet.py). A combination of the two might be interesting to see!

Right now video prediction is hard to control as the new frames are highly dependent on the prompt, if we could use images we would have better/finer control. This pipeline would enable apps like Blender to generate new images based on past reference frames and a depth buffer.

Looking at the code it looks like this is doable but before I try, I would like to get the input and suggestions of more expert people on this possible approach (@a-r-r-o-w or @DN6 :) ):

1. make [pipeline_animatediff_sparsectrl.py](https://github.com/huggingface/diffusers/blob/e417d028115e72b953a73e39d9687aa70ba3e37e/src/diffusers/pipelines/animatediff/pipeline_animatediff_sparsectrl.py) and [pipeline_animatediff_controlnet.py](https://github.com/huggingface/diffusers/blob/e417d028115e72b953a73e39d9687aa70ba3e37e/src/diffusers/pipelines/animatediff/pipeline_animatediff_controlnet.py) as similar as possible so diffing shows as much as common code as possible
2. refactor the blocks of code that are different into functions
3. have these functions work together in a new single pipeline

Does this make sense?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.