huggingface / huggingface/diffusers

Add support for PPVCtrl

Open
#10,625 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

### Model/Pipeline/Scheduler description

I recently came across an impressive work called PPVCtrl, a controllable video generation model. It leverages an auxiliary condition encoder to transform a text-to-video generation model into a customizable video generator, all without retraining the original generator. It's akin to ControlNet, but for video generation.



Prompt
Reference Image
Control Videos
PP-VCtrl-5B-T2V
PP-VCtrl-5B-I2V




Group of fishes swimming in aquarium.
Reference
Conrotl Videos
PP-VCtrl T2V
PP-VCtrl I2V


A boat with a flag on it is sailing on the sea.
Reference
Control Videos
PP-VCtrl T2v
PP-VCtrl I2v




Prompt
Reference Image
Control Videos
PP-VCtrl-5B-T2V
PP-VCtrl-5B-I2V




A rider in a dark helmet and white breeches is atop a chestnut horse...
Reference
Conrotl Videos
PP-VCtrl T2V
PP-VCtrl I2V


A dark gray Mini Cooper is parked on a city street...
Reference
Control Videos
PP-VCtrl T2v
PP-VCtrl I2v




Prompt
Reference Image
Pose Videos
PP-VCtrl-5B-I2V




A young man with curly hair and a red t-shirt featuring a white logo is seen in various states of motion...

Reference 1


Pose Videos


PP-VCtrl 1



A woman models an Adrianna Papell women's gown, featuring a sleeveless...

Reference 2


Pose Videos


PP-VCtrl 2


### Open source status

- [x] The model implementation is available.
- [x] The model weights are available (Only relevant if addition is not a scheduler).

### Provide useful links for the implementation

https://github.com/PaddlePaddle/PaddleMIX/tree/develop/ppdiffusers/examples/ppvctrl

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.