huggingface / huggingface/diffusers

[modular] `guider` should be an optional component for text encoder blocks

Aperta
#14,469 0 commenti 0 reazioni 1 assegnatario Rivendicata da @yiyixuxu Vedi su GitHub
feature-request modular-pipelines
Lingua principale
Python
Stelle
34.5k
Fork
7.3k
Merge medio
3g 3h
PR unite (30g)
91

Descrizione

Most text encoder blocks declare the guider as an expected component and decide from it whether to produce negative embeddings. We've started moving away from that — `WanAnimate2TextEncoderStep` doesn't declare one — but the codebase hasn't converged, so I want to write the rule down.

The guider should be **optional** for a text encoder block, so it behaves sensibly in both capacities:

**Standalone** — the block should not be aware of the guider at all. If a user passes a `negative_prompt`, they get `negative_prompt_embeds` back. Today they may not: the block asks the guider, and if that pipeline's guider spec happens to default to disabled, the negative prompt is silently dropped. That's not intuitive — you asked for something and got nothing back, for a reason belonging to a component you never used (see #13505).

**Combined into the pipeline, with the denoiser** — here the block *should* be aware of the guider setting: with the guider disabled it should not produce negative embeddings (nothing would consume them), and it should **warn** when a `negative_prompt` was supplied, so the user learns why it had no effect instead of silently getting nothing back.

`wan_animate_2` is closest to this today, but it encodes the negative prompt anyway rather than warning and skipping, so even that one needs a pass:

```python
# src/diffusers/modular_pipelines/wan_animate_2/encoders.py
block_state.negative_prompt_embeds = None
if components.requires_unconditional_embeds or block_state.negative_prompt is not None:
...
```

Note `ComponentSpec` has no notion of an optional component today — declaring it means the block expects it — so part of this is deciding whether we formalize that, or keep the wan_animate_2 shape (don't declare it; consult `requires_unconditional_embeds`, which is `False` when no guider exists).

**State of the codebase**, so we know the size of the pass:

- declares a `guider` spec, reads `components.requires_unconditional_embeds` — `ernie_image`, `flux2`, `helios`, `hunyuan_video1_5`, `krea2`, `ltx`, `qwenimage`, `wan`, `z_image`
- declares a `guider` spec, reads it directly (`components.guider.num_conditions > 1`) — `anima`, `stable_diffusion_xl`
- no guider in the text encoder — `wan_animate_2` (by design), `ltx2` and `stable_diffusion_3` (denoise-only)

plus `requires_unconditional_embeds` copy-pasted into ten `ModularPipeline` subclasses with identical bodies (`flux2` adds an `is_distilled` early return). Once settled it goes into `.ai/modular.md` — this came up writing those docs in #14452.

> Opened by Claude (Opus 5) on behalf of @yiyixuxu.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.