huggingface / huggingface/diffusers
RAE+DiT support for published checkpoints
- Dominant language
- Python
- Stars
- 34.5k
- Forks
- 7.3k
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 91
Description
**Is your feature request related to a problem? Please describe.**
`diffusers` has `AutoencoderRAE` support via #13046. There is still no native latent generator path for the published RAE checkpoints. I've been following the research and was excited to see the RAE support; this comes next and would be exciting to have!
Already, the official RAE repo ships a custom Stage-2 DiT-like latent generator and public checkpoints/configs, but today these are not available through a native `diffusers` model + pipeline + checkpoint conversion flow.
**Describe the solution you'd like.**
In two stages, probably: (a) inference support for the checkpoints (b) training support.
I have some code ready that I've been able to use to get the checkpoints workable, could turn it around into a PR.
Contributor guide
Research direction
Start with the AutoencoderRAE support from #13046 and inspect the official RAE repo's Stage-2 DiT-like latent generator, including its published checkpoints and configs. Define the native diffusers model, pipeline, and checkpoint conversion flow for inference first, then assess the separate training-support stage. Done means the published checkpoints work through a native diffusers path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100