huggingface / huggingface/diffusers

Community pipeline: rerank Stable Diffusion candidates by CLIP score

Open
#14,721 0 comments 0 reactions 0 assignees View on GitHub
feature-request pipelines
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

A single sample from a txt2img pipeline can miss the prompt (wrong count, dropped attribute, ignored token) even when the average output at that seed/guidance combo is fine. I put together a community pipeline that draws N candidates for one prompt, scores each one against the prompt with a CLIP model, and returns them sorted best-first.

It wraps a plain `StableDiffusionPipeline` built from the same components you'd pass to the regular one, so it works with whatever UNet/VAE/scheduler combination that pipeline already accepts. It doesn't touch the denoising trajectory itself, unlike `CLIPGuidedStableDiffusion`, which steers sampling with CLIP gradients as it goes. This one just generates independently and picks.

No GPU on the box I'm on, so I verified it against the tiny fixtures diffusers' own test suite uses (`hf-internal-testing/tiny-stable-diffusion-torch` + a CLIP model built from `tiny-random-clip`'s own config) rather than a full-size checkpoint. That confirms the plumbing, not image quality:

```
num images returned: 4
clip_scores (best-first): [1.9110276699066162, 1.8932260274887085, 1.8369966745376587, 1.784890055656433]
default return_all=False -> images: 1 scores: [3.9261884689331055]
generator/num_candidates mismatch correctly rejected: Got 1 generators for num_candidates=2.
batched prompt correctly rejected: CLIPRerankStableDiffusionPipeline reranks candidates for one prompt at a time; call it once per prompt if you have a batch.
OK: all checks passed
```

Also ran `ruff check` and `ruff format --check` clean against the file.

One open question: would you rather this lived as a method/flag on the existing pipeline (`return_dict`-style opt-in) than a separate community pipeline class? I went with a separate class since that's the shape the rest of `examples/community` uses, but a rerank option is a smaller surface if that's preferred.

Happy to open a PR with this, including a couple of tests against the same tiny fixtures.

Contributor guide

Open the contributing guide

Research direction

Review the conventions in examples/community, along with StableDiffusionPipeline and CLIPGuidedStableDiffusion, before deciding whether this belongs as a separate pipeline or an existing-pipeline option. Reproduce the tiny Stable Diffusion and tiny-random-clip checks described in the issue; done means candidates are scored and sorted, return_all behavior works, invalid generator counts and batched prompts are rejected, and the mentioned ruff checks remain clean.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.