huggingface / huggingface/diffusers

Community pipeline: rerank Stable Diffusion candidates by CLIP score

Aperta
#14,721 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
feature-request pipelines
Lingua principale
Python
Stelle
34.5k
Fork
7.3k
Merge medio
3g 3h
PR unite (30g)
91

Descrizione

A single sample from a txt2img pipeline can miss the prompt (wrong count, dropped attribute, ignored token) even when the average output at that seed/guidance combo is fine. I put together a community pipeline that draws N candidates for one prompt, scores each one against the prompt with a CLIP model, and returns them sorted best-first.

It wraps a plain `StableDiffusionPipeline` built from the same components you'd pass to the regular one, so it works with whatever UNet/VAE/scheduler combination that pipeline already accepts. It doesn't touch the denoising trajectory itself, unlike `CLIPGuidedStableDiffusion`, which steers sampling with CLIP gradients as it goes. This one just generates independently and picks.

No GPU on the box I'm on, so I verified it against the tiny fixtures diffusers' own test suite uses (`hf-internal-testing/tiny-stable-diffusion-torch` + a CLIP model built from `tiny-random-clip`'s own config) rather than a full-size checkpoint. That confirms the plumbing, not image quality:

```
num images returned: 4
clip_scores (best-first): [1.9110276699066162, 1.8932260274887085, 1.8369966745376587, 1.784890055656433]
default return_all=False -> images: 1 scores: [3.9261884689331055]
generator/num_candidates mismatch correctly rejected: Got 1 generators for num_candidates=2.
batched prompt correctly rejected: CLIPRerankStableDiffusionPipeline reranks candidates for one prompt at a time; call it once per prompt if you have a batch.
OK: all checks passed
```

Also ran `ruff check` and `ruff format --check` clean against the file.

One open question: would you rather this lived as a method/flag on the existing pipeline (`return_dict`-style opt-in) than a separate community pipeline class? I went with a separate class since that's the shape the rest of `examples/community` uses, but a rerank option is a smaller surface if that's preferred.

Happy to open a PR with this, including a couple of tests against the same tiny fixtures.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Review the conventions in examples/community, along with StableDiffusionPipeline and CLIPGuidedStableDiffusion, before deciding whether this belongs as a separate pipeline or an existing-pipeline option. Reproduce the tiny Stable Diffusion and tiny-random-clip checks described in the issue; done means candidates are scored and sorted, return_all behavior works, invalid generator counts and batched prompts are rejected, and the mentioned ruff checks remain clean.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
huggingface, python, pytorch
Ambito
machine-learning
Tipo di issue
Funzionalità
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Attiva
Chiarezza
Abbastanza chiara
Idoneità per principianti
55/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.