huggingface / huggingface/diffusers

RAE+DiT support for published checkpoints

Open
#13,225 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

**Is your feature request related to a problem? Please describe.**
`diffusers` has `AutoencoderRAE` support via #13046. There is still no native latent generator path for the published RAE checkpoints. I've been following the research and was excited to see the RAE support; this comes next and would be exciting to have!

Already, the official RAE repo ships a custom Stage-2 DiT-like latent generator and public checkpoints/configs, but today these are not available through a native `diffusers` model + pipeline + checkpoint conversion flow.

**Describe the solution you'd like.**
In two stages, probably: (a) inference support for the checkpoints (b) training support.
I have some code ready that I've been able to use to get the checkpoints workable, could turn it around into a PR.

Contributor guide

Open the contributing guide

Research direction

Start with the AutoencoderRAE support from #13046 and inspect the official RAE repo's Stage-2 DiT-like latent generator, including its published checkpoints and configs. Define the native diffusers model, pipeline, and checkpoint conversion flow for inference first, then assess the separate training-support stage. Done means the published checkpoints work through a native diffusers path.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.