huggingface / huggingface/diffusers
Community pipeline: rerank Stable Diffusion candidates by CLIP score
- Vorherrschende Sprache
- Python
- Sterne
- 34.5k
- Forks
- 7.3k
- Ø Merge
- 3 T. 3 Std.
- Gemergte PRs (30 T.)
- 91
Beschreibung
A single sample from a txt2img pipeline can miss the prompt (wrong count, dropped attribute, ignored token) even when the average output at that seed/guidance combo is fine. I put together a community pipeline that draws N candidates for one prompt, scores each one against the prompt with a CLIP model, and returns them sorted best-first.
It wraps a plain `StableDiffusionPipeline` built from the same components you'd pass to the regular one, so it works with whatever UNet/VAE/scheduler combination that pipeline already accepts. It doesn't touch the denoising trajectory itself, unlike `CLIPGuidedStableDiffusion`, which steers sampling with CLIP gradients as it goes. This one just generates independently and picks.
No GPU on the box I'm on, so I verified it against the tiny fixtures diffusers' own test suite uses (`hf-internal-testing/tiny-stable-diffusion-torch` + a CLIP model built from `tiny-random-clip`'s own config) rather than a full-size checkpoint. That confirms the plumbing, not image quality:
```
num images returned: 4
clip_scores (best-first): [1.9110276699066162, 1.8932260274887085, 1.8369966745376587, 1.784890055656433]
default return_all=False -> images: 1 scores: [3.9261884689331055]
generator/num_candidates mismatch correctly rejected: Got 1 generators for num_candidates=2.
batched prompt correctly rejected: CLIPRerankStableDiffusionPipeline reranks candidates for one prompt at a time; call it once per prompt if you have a batch.
OK: all checks passed
```
Also ran `ruff check` and `ruff format --check` clean against the file.
One open question: would you rather this lived as a method/flag on the existing pipeline (`return_dict`-style opt-in) than a separate community pipeline class? I went with a separate class since that's the shape the rest of `examples/community` uses, but a rerank option is a smaller surface if that's preferred.
Happy to open a PR with this, including a couple of tests against the same tiny fixtures.
Beitragsleitfaden
Rechercherichtung
Prüfe die Konventionen in examples/community sowie StableDiffusionPipeline und CLIPGuidedStableDiffusion, bevor du entscheidest, ob dies als separate Pipeline oder als Option einer bestehenden Pipeline umgesetzt werden sollte. Stelle die in der Issue beschriebenen tiny Stable Diffusion- und tiny-random-clip-Prüfungen nach; als erledigt gilt dies, wenn Kandidaten bewertet und sortiert werden, das Verhalten von return_all funktioniert, ungültige Anzahlen von Generatoren und gebatchte Prompts abgewiesen werden und die genannten ruff-Prüfungen weiterhin sauber bleiben.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- huggingface, python, pytorch
- Bereich
- machine-learning
- Issue-Typ
- Feature
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Aktivitätsstatus
- Aktiv
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 55/100