huggingface / huggingface/diffusers
Implementing training-free RB-Modulation pipeline for most used models
- Dominant language
- Python
- Stars
- 34.5k
- Forks
- 7.3k
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 91
Description
### Model/Pipeline/Scheduler description
The [RB-Modulation algorithm](https://rb-modulation.github.io/) is **training-free** technique to produce image 2 image style and content transfer in diffusion model. It has two components:
1. Stochastic Optimization Control (SOC): This component requires an [evaluator](https://github.com/learn2phoenix/CSD) for the style at each timestep. Therefore, an evaluator model and control function pipeline has to be built.
2. AttentionFeatureAggregation (AFA): This needs a clip image encoder to concat the K,V features of the image and caption. A slight tweak has to be done in the forward pass of the existing models.
This will be an interesting implementation for edits as the paper shows promising results.
### Open source status
- [X] The model implementation is available.
- [X] The model weights are available (Only relevant if addition is not a scheduler).
### Provide useful links for the implementation
### RB-Modulation:
Title: RB-Modulation: Training-Free Personalization of Diffusion Models using Stochastic Optimal Control
Code Link: https://github.com/google/RB-Modulation
Authors: Litu Rout and Yujia Chen and Nataniel Ruiz and Abhishek Kumar and Constantine Caramanis and Sanjay Shakkottai and Wen-Sheng Chu
Authors GH Username: @LituRout, @IssacCyj
### Style Evaluator:
Title: Measuring Style Similarity in Diffusion Models
Code Link: https://github.com/learn2phoenix/CSD
Authors: Somepalli, Gowthami and Gupta, Anubhav and Gupta, Kamal and Palta, Shramay and Goldblum, Micah and Geiping, Jonas and Shrivastava, Abhinav and Goldstein, Tom
Authors Username: @somepago, @learn2phoenix
Contributor guide
Assessment
This issue has not been assessed yet.