huggingface / huggingface/diffusers
Community pipeline: rerank Stable Diffusion candidates by CLIP score
- Lingua principale
- Python
- Stelle
- 34.5k
- Fork
- 7.3k
- Merge medio
- 3g 3h
- PR unite (30g)
- 91
Descrizione
A single sample from a txt2img pipeline can miss the prompt (wrong count, dropped attribute, ignored token) even when the average output at that seed/guidance combo is fine. I put together a community pipeline that draws N candidates for one prompt, scores each one against the prompt with a CLIP model, and returns them sorted best-first.
It wraps a plain `StableDiffusionPipeline` built from the same components you'd pass to the regular one, so it works with whatever UNet/VAE/scheduler combination that pipeline already accepts. It doesn't touch the denoising trajectory itself, unlike `CLIPGuidedStableDiffusion`, which steers sampling with CLIP gradients as it goes. This one just generates independently and picks.
No GPU on the box I'm on, so I verified it against the tiny fixtures diffusers' own test suite uses (`hf-internal-testing/tiny-stable-diffusion-torch` + a CLIP model built from `tiny-random-clip`'s own config) rather than a full-size checkpoint. That confirms the plumbing, not image quality:
```
num images returned: 4
clip_scores (best-first): [1.9110276699066162, 1.8932260274887085, 1.8369966745376587, 1.784890055656433]
default return_all=False -> images: 1 scores: [3.9261884689331055]
generator/num_candidates mismatch correctly rejected: Got 1 generators for num_candidates=2.
batched prompt correctly rejected: CLIPRerankStableDiffusionPipeline reranks candidates for one prompt at a time; call it once per prompt if you have a batch.
OK: all checks passed
```
Also ran `ruff check` and `ruff format --check` clean against the file.
One open question: would you rather this lived as a method/flag on the existing pipeline (`return_dict`-style opt-in) than a separate community pipeline class? I went with a separate class since that's the shape the rest of `examples/community` uses, but a rerank option is a smaller surface if that's preferred.
Happy to open a PR with this, including a couple of tests against the same tiny fixtures.
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Review the conventions in examples/community, along with StableDiffusionPipeline and CLIPGuidedStableDiffusion, before deciding whether this belongs as a separate pipeline or an existing-pipeline option. Reproduce the tiny Stable Diffusion and tiny-random-clip checks described in the issue; done means candidates are scored and sorted, return_all behavior works, invalid generator counts and batched prompts are rejected, and the mentioned ruff checks remain clean.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- huggingface, python, pytorch
- Ambito
- machine-learning
- Tipo di issue
- Funzionalità
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Attiva
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 55/100