huggingface / huggingface/diffusers

[Pipeline] ReNoise: Real Image Inversion Through Iterative Noising

Open
#7,509 6 comments 0 reactions 0 assignees View on GitHub
community-examples contributions-welcome stale
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

### Model/Pipeline/Scheduler description

Achieving faithful image-to-noise inversion with Denoising Diffusion models remains a challenge, particularly for more recent models trained to generate images with a small number of denoising steps. This work introduces an inversion method with a high quality-to-operation ratio, enhancing reconstruction accuracy without increasing the number of operations. Building on reversing the diffusion sampling process, the method applies multiple fixed-point iterations to estimate the next inversion target at each noise level, and then averages the predictions to empirically increase image reconstruction quality. Furthermore, this method preserves editability through optimizing noise regularization losses in a fashion similar to Pix2PixZero.

### Open source status

- [ ] The model implementation is available.
- [ ] The model weights are available (Only relevant if addition is not a scheduler).

### Provide useful links for the implementation

- Arxiv PDF: https://arxiv.org/pdf/2403.14602.pdf
- Github: https://github.com/garibida/ReNoise-Inversion
- First Author: @garibida
- Notes: The code is not available yet, but the method should be easy to implement as a new pipeline similar to the existing `StableDiffusionPix2PixZeroPipeline` based on the pseudocode and discussions in the paper.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.