Implementing training-free RB-Modulation pipeline for most used models

Model/Pipeline/Scheduler description

The RB-Modulation algorithm is training-free technique to produce image 2 image style and content transfer in diffusion model. It has two components:

1. Stochastic Optimization Control (SOC): This component requires an evaluator for the style at each timestep. Therefore, an evaluator model and control function pipeline has to be built.
2. AttentionFeatureAggregation (AFA): This needs a clip image encoder to concat the K,V features of the image and caption. A slight tweak has to be done in the forward pass of the existing models.

This will be an interesting implementation for edits as the paper shows promising results.

Open source status

• The model implementation is available.
• The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

RB-Modulation:

Title: RB-Modulation: Training-Free Personalization of Diffusion Models using Stochastic Optimal Control
Code Link: https://github.com/google/RB-Modulation
Authors: Litu Rout and Yujia Chen and Nataniel Ruiz and Abhishek Kumar and Constantine Caramanis and Sanjay Shakkottai and Wen-Sheng Chu
Authors GH Username: @LituRout, @IssacCyj

Style Evaluator:

Title: Measuring Style Similarity in Diffusion Models
Code Link: https://github.com/learn2phoenix/CSD
Authors: Somepalli, Gowthami and Gupta, Anubhav and Gupta, Kamal and Palta, Shramay and Goldblum, Micah and Geiping, Jonas and Shrivastava, Abhinav and Goldstein, Tom
Authors Username: @somepago, @learn2phoenix

Comienza revisando la implementación y el paper de RB-Modulation, y después compara los puntos de entrada existentes del modelo de difusión y del scheduler en el repositorio con los componentes SOC y AttentionFeatureAggregation descritos aquí. Investiga el evaluador CSD y el código enlazado de Google/RB-Modulation antes de definir los modelos compatibles y los límites de integración. Se considera terminado cuando exista una canalización funcional y sin entrenamiento para la transferencia de estilo y contenido de imagen a imagen para los modelos seleccionados.