huggingface / huggingface/diffusers
Community pipeline: rerank Stable Diffusion candidates by CLIP score
- Dominant language
- Python
- Stars
- 34.5k
- Forks
- 7.3k
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 91
Description
A single sample from a txt2img pipeline can miss the prompt (wrong count, dropped attribute, ignored token) even when the average output at that seed/guidance combo is fine. I put together a community pipeline that draws N candidates for one prompt, scores each one against the prompt with a CLIP model, and returns them sorted best-first.
It wraps a plain `StableDiffusionPipeline` built from the same components you'd pass to the regular one, so it works with whatever UNet/VAE/scheduler combination that pipeline already accepts. It doesn't touch the denoising trajectory itself, unlike `CLIPGuidedStableDiffusion`, which steers sampling with CLIP gradients as it goes. This one just generates independently and picks.
No GPU on the box I'm on, so I verified it against the tiny fixtures diffusers' own test suite uses (`hf-internal-testing/tiny-stable-diffusion-torch` + a CLIP model built from `tiny-random-clip`'s own config) rather than a full-size checkpoint. That confirms the plumbing, not image quality:
```
num images returned: 4
clip_scores (best-first): [1.9110276699066162, 1.8932260274887085, 1.8369966745376587, 1.784890055656433]
default return_all=False -> images: 1 scores: [3.9261884689331055]
generator/num_candidates mismatch correctly rejected: Got 1 generators for num_candidates=2.
batched prompt correctly rejected: CLIPRerankStableDiffusionPipeline reranks candidates for one prompt at a time; call it once per prompt if you have a batch.
OK: all checks passed
```
Also ran `ruff check` and `ruff format --check` clean against the file.
One open question: would you rather this lived as a method/flag on the existing pipeline (`return_dict`-style opt-in) than a separate community pipeline class? I went with a separate class since that's the shape the rest of `examples/community` uses, but a rerank option is a smaller surface if that's preferred.
Happy to open a PR with this, including a couple of tests against the same tiny fixtures.
Contributor guide
Research direction
Review the conventions in examples/community, along with StableDiffusionPipeline and CLIPGuidedStableDiffusion, before deciding whether this belongs as a separate pipeline or an existing-pipeline option. Reproduce the tiny Stable Diffusion and tiny-random-clip checks described in the issue; done means candidates are scored and sorted, return_all behavior works, invalid generator counts and batched prompts are rejected, and the mentioned ruff checks remain clean.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100