huggingface / huggingface/diffusers
[modular] `guider` should be an optional component for text encoder blocks
- Vorherrschende Sprache
- Python
- Sterne
- 34.5k
- Forks
- 7.3k
- Ø Merge
- 3 T. 3 Std.
- Gemergte PRs (30 T.)
- 91
Beschreibung
Most text encoder blocks declare the guider as an expected component and decide from it whether to produce negative embeddings. We've started moving away from that — `WanAnimate2TextEncoderStep` doesn't declare one — but the codebase hasn't converged, so I want to write the rule down.
The guider should be **optional** for a text encoder block, so it behaves sensibly in both capacities:
**Standalone** — the block should not be aware of the guider at all. If a user passes a `negative_prompt`, they get `negative_prompt_embeds` back. Today they may not: the block asks the guider, and if that pipeline's guider spec happens to default to disabled, the negative prompt is silently dropped. That's not intuitive — you asked for something and got nothing back, for a reason belonging to a component you never used (see #13505).
**Combined into the pipeline, with the denoiser** — here the block *should* be aware of the guider setting: with the guider disabled it should not produce negative embeddings (nothing would consume them), and it should **warn** when a `negative_prompt` was supplied, so the user learns why it had no effect instead of silently getting nothing back.
`wan_animate_2` is closest to this today, but it encodes the negative prompt anyway rather than warning and skipping, so even that one needs a pass:
```python
# src/diffusers/modular_pipelines/wan_animate_2/encoders.py
block_state.negative_prompt_embeds = None
if components.requires_unconditional_embeds or block_state.negative_prompt is not None:
...
```
Note `ComponentSpec` has no notion of an optional component today — declaring it means the block expects it — so part of this is deciding whether we formalize that, or keep the wan_animate_2 shape (don't declare it; consult `requires_unconditional_embeds`, which is `False` when no guider exists).
**State of the codebase**, so we know the size of the pass:
- declares a `guider` spec, reads `components.requires_unconditional_embeds` — `ernie_image`, `flux2`, `helios`, `hunyuan_video1_5`, `krea2`, `ltx`, `qwenimage`, `wan`, `z_image`
- declares a `guider` spec, reads it directly (`components.guider.num_conditions > 1`) — `anima`, `stable_diffusion_xl`
- no guider in the text encoder — `wan_animate_2` (by design), `ltx2` and `stable_diffusion_3` (denoise-only)
plus `requires_unconditional_embeds` copy-pasted into ten `ModularPipeline` subclasses with identical bodies (`flux2` adds an `is_distilled` early return). Once settled it goes into `.ai/modular.md` — this came up writing those docs in #14452.
> Opened by Claude (Opus 5) on behalf of @yiyixuxu.
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.