huggingface / huggingface/diffusers
Add SeFi-Image pipeline
- Lingua principale
- Python
- Stelle
- 34.5k
- Fork
- 7.3k
- Merge medio
- 3g 3h
- PR unite (30g)
- 91
Descrizione
### Model/Pipeline/Scheduler description
We are proposing the integration of SeFi-Image into Diffusers.
SeFi-Image is a text-to-image model family built with Semantic-First Diffusion. It separates image generation into semantic and texture latent streams, denoising the semantic structure slightly ahead of texture details so the texture stream receives a cleaner structural anchor during generation.
Key characteristics:
- Semantic-first generation: separates semantic and texture latents and advances the semantic denoising stream to improve structural consistency.
- Generation-reconstruction trade-off: combines a compact semantic latent for easier generation with a high-fidelity texture latent for reconstruction detail.
- Multiple model scales and variants: 1B/2B/5B Base checkpoints, 5B RL checkpoint, and 1B/2B/5B Turbo checkpoints.
- Few-step Turbo inference: Turbo checkpoints default to 4 denoising steps with guidance scale 1.0.
- Standard text-to-image usage: prompt-to-image generation at 1024x1024 by default.
The proposed Diffusers integration adds:
- `SeFiTransformer2DModel`
- `SeFiPipeline`
- an original-checkpoint conversion script
- API documentation
- fast model and pipeline tests
The current implementation was validated with the real `SeFi-Image/SeFi-Image-1B-turbo` checkpoint, including conversion, 1024x1024 inference parity against a reference output, and CPU offload smoke testing.
### Open source status
- [x] The model implementation is available.
- [x] The model weights are available (Only relevant if addition is not a scheduler).
### Provide useful links for the implementation
- GitHub repository: https://github.com/jmliu206/SeFi-Image
- Project page: https://jmliu206.github.io/sefi-web/
- Technical report: https://arxiv.org/pdf/2606.22568
- Hugging Face organization / model collection: https://huggingface.co/SeFi-Image
- Example checkpoints:
- https://huggingface.co/SeFi-Image/SeFi-Image-1B-Base
- https://huggingface.co/SeFi-Image/SeFi-Image-2B-Base
- https://huggingface.co/SeFi-Image/SeFi-Image-5B-Base
- https://huggingface.co/SeFi-Image/SeFi-Image-5B-RL
- https://huggingface.co/SeFi-Image/SeFi-Image-1B-turbo
- https://huggingface.co/SeFi-Image/SeFi-Image-2B-turbo
- https://huggingface.co/SeFi-Image/SeFi-Image-5B-turbo
- Diffusers implementation PR: https://github.com/huggingface/diffusers/pull/14084
### Additional context
This is an official integration proposal from the SeFi-Image authors/maintainers. The draft PR is already open with the implementation and validation results, and can be adjusted based on maintainer feedback on the desired scope or API shape.
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Inizia con la PR di implementazione #14084 e il repository SeFi-Image per comprendere il SeFiTransformer2DModel proposto, SeFiPipeline, lo script di conversione, la documentazione e i test. Il lavoro è completato quando l'integrazione viene accettata con la conversione, la parità dell'output di riferimento 1024x1024, uno smoke test del CPU offload e test rapidi del modello e della pipeline.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- python, pytorch
- Ambito
- machine-learning
- Tipo di issue
- Funzionalità
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Stato di attività
- Ferma
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 25/100