Implementing training-free RB-Modulation pipeline for most used models

Model/Pipeline/Scheduler description

The RB-Modulation algorithm is training-free technique to produce image 2 image style and content transfer in diffusion model. It has two components:

1. Stochastic Optimization Control (SOC): This component requires an evaluator for the style at each timestep. Therefore, an evaluator model and control function pipeline has to be built.
2. AttentionFeatureAggregation (AFA): This needs a clip image encoder to concat the K,V features of the image and caption. A slight tweak has to be done in the forward pass of the existing models.

This will be an interesting implementation for edits as the paper shows promising results.

Open source status

• The model implementation is available.
• The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

RB-Modulation:

Title: RB-Modulation: Training-Free Personalization of Diffusion Models using Stochastic Optimal Control
Code Link: https://github.com/google/RB-Modulation
Authors: Litu Rout and Yujia Chen and Nataniel Ruiz and Abhishek Kumar and Constantine Caramanis and Sanjay Shakkottai and Wen-Sheng Chu
Authors GH Username: @LituRout, @IssacCyj

Style Evaluator:

Title: Measuring Style Similarity in Diffusion Models
Code Link: https://github.com/learn2phoenix/CSD
Authors: Somepalli, Gowthami and Gupta, Anubhav and Gupta, Kamal and Palta, Shramay and Goldblum, Micah and Geiping, Jonas and Shrivastava, Abhinav and Goldstein, Tom
Authors Username: @somepago, @learn2phoenix

まず RB-Modulation の実装と paper を確認し、次にリポジトリにある既存の diffusion model と scheduler のエントリポイントを、ここで説明されている SOC および AttentionFeatureAggregation コンポーネントと比較します。対応するモデルと統合境界を定義する前に、CSD evaluator とリンク先の Google/RB-Modulation コードを調査します。選択したモデルで、トレーニング不要の image-to-image によるスタイルおよびコンテンツ転送パイプラインが動作すれば完了です。