huggingface / huggingface/diffusers

Implementing training-free RB-Modulation pipeline for most used models

Open
#9,283 4 comments 2 reactions 0 assignees View on GitHub
consider-for-modular-diffusers wip
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

### Model/Pipeline/Scheduler description

The [RB-Modulation algorithm](https://rb-modulation.github.io/) is **training-free** technique to produce image 2 image style and content transfer in diffusion model. It has two components:
1. Stochastic Optimization Control (SOC): This component requires an [evaluator](https://github.com/learn2phoenix/CSD) for the style at each timestep. Therefore, an evaluator model and control function pipeline has to be built.
2. AttentionFeatureAggregation (AFA): This needs a clip image encoder to concat the K,V features of the image and caption. A slight tweak has to be done in the forward pass of the existing models.

This will be an interesting implementation for edits as the paper shows promising results.

### Open source status

- [X] The model implementation is available.
- [X] The model weights are available (Only relevant if addition is not a scheduler).

### Provide useful links for the implementation

### RB-Modulation:
Title: RB-Modulation: Training-Free Personalization of Diffusion Models using Stochastic Optimal Control
Code Link: https://github.com/google/RB-Modulation
Authors: Litu Rout and Yujia Chen and Nataniel Ruiz and Abhishek Kumar and Constantine Caramanis and Sanjay Shakkottai and Wen-Sheng Chu
Authors GH Username: @LituRout, @IssacCyj

### Style Evaluator:
Title: Measuring Style Similarity in Diffusion Models
Code Link: https://github.com/learn2phoenix/CSD
Authors: Somepalli, Gowthami and Gupta, Anubhav and Gupta, Kamal and Palta, Shramay and Goldblum, Micah and Geiping, Jonas and Shrivastava, Abhinav and Goldstein, Tom
Authors Username: @somepago, @learn2phoenix

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.