huggingface / huggingface/diffusers

Add SeFi-Image pipeline

Aperta
#14,086 0 commenti 1 reazione 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
34.5k
Fork
7.3k
Merge medio
3g 3h
PR unite (30g)
91

Descrizione

### Model/Pipeline/Scheduler description

We are proposing the integration of SeFi-Image into Diffusers.

SeFi-Image is a text-to-image model family built with Semantic-First Diffusion. It separates image generation into semantic and texture latent streams, denoising the semantic structure slightly ahead of texture details so the texture stream receives a cleaner structural anchor during generation.

Key characteristics:

- Semantic-first generation: separates semantic and texture latents and advances the semantic denoising stream to improve structural consistency.
- Generation-reconstruction trade-off: combines a compact semantic latent for easier generation with a high-fidelity texture latent for reconstruction detail.
- Multiple model scales and variants: 1B/2B/5B Base checkpoints, 5B RL checkpoint, and 1B/2B/5B Turbo checkpoints.
- Few-step Turbo inference: Turbo checkpoints default to 4 denoising steps with guidance scale 1.0.
- Standard text-to-image usage: prompt-to-image generation at 1024x1024 by default.

The proposed Diffusers integration adds:

- `SeFiTransformer2DModel`
- `SeFiPipeline`
- an original-checkpoint conversion script
- API documentation
- fast model and pipeline tests

The current implementation was validated with the real `SeFi-Image/SeFi-Image-1B-turbo` checkpoint, including conversion, 1024x1024 inference parity against a reference output, and CPU offload smoke testing.

### Open source status

- [x] The model implementation is available.
- [x] The model weights are available (Only relevant if addition is not a scheduler).

### Provide useful links for the implementation

- GitHub repository: https://github.com/jmliu206/SeFi-Image
- Project page: https://jmliu206.github.io/sefi-web/
- Technical report: https://arxiv.org/pdf/2606.22568
- Hugging Face organization / model collection: https://huggingface.co/SeFi-Image
- Example checkpoints:
- https://huggingface.co/SeFi-Image/SeFi-Image-1B-Base
- https://huggingface.co/SeFi-Image/SeFi-Image-2B-Base
- https://huggingface.co/SeFi-Image/SeFi-Image-5B-Base
- https://huggingface.co/SeFi-Image/SeFi-Image-5B-RL
- https://huggingface.co/SeFi-Image/SeFi-Image-1B-turbo
- https://huggingface.co/SeFi-Image/SeFi-Image-2B-turbo
- https://huggingface.co/SeFi-Image/SeFi-Image-5B-turbo
- Diffusers implementation PR: https://github.com/huggingface/diffusers/pull/14084

### Additional context

This is an official integration proposal from the SeFi-Image authors/maintainers. The draft PR is already open with the implementation and validation results, and can be adjusted based on maintainer feedback on the desired scope or API shape.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia con la PR di implementazione #14084 e il repository SeFi-Image per comprendere il SeFiTransformer2DModel proposto, SeFiPipeline, lo script di conversione, la documentazione e i test. Il lavoro è completato quando l'integrazione viene accettata con la conversione, la parità dell'output di riferimento 1024x1024, uno smoke test del CPU offload e test rapidi del modello e della pipeline.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python, pytorch
Ambito
machine-learning
Tipo di issue
Funzionalità
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
25/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.