huggingface / huggingface/diffusers

[modular] `guider` should be an optional component for text encoder blocks

Offen
#14,469 0 Kommentare 0 Reaktionen 1 zugewiesene Person Beansprucht von @yiyixuxu Auf GitHub ansehen
feature-request modular-pipelines
Vorherrschende Sprache
Python
Sterne
34.5k
Forks
7.3k
Ø Merge
3 T. 3 Std.
Gemergte PRs (30 T.)
91

Beschreibung

Most text encoder blocks declare the guider as an expected component and decide from it whether to produce negative embeddings. We've started moving away from that — `WanAnimate2TextEncoderStep` doesn't declare one — but the codebase hasn't converged, so I want to write the rule down.

The guider should be **optional** for a text encoder block, so it behaves sensibly in both capacities:

**Standalone** — the block should not be aware of the guider at all. If a user passes a `negative_prompt`, they get `negative_prompt_embeds` back. Today they may not: the block asks the guider, and if that pipeline's guider spec happens to default to disabled, the negative prompt is silently dropped. That's not intuitive — you asked for something and got nothing back, for a reason belonging to a component you never used (see #13505).

**Combined into the pipeline, with the denoiser** — here the block *should* be aware of the guider setting: with the guider disabled it should not produce negative embeddings (nothing would consume them), and it should **warn** when a `negative_prompt` was supplied, so the user learns why it had no effect instead of silently getting nothing back.

`wan_animate_2` is closest to this today, but it encodes the negative prompt anyway rather than warning and skipping, so even that one needs a pass:

```python
# src/diffusers/modular_pipelines/wan_animate_2/encoders.py
block_state.negative_prompt_embeds = None
if components.requires_unconditional_embeds or block_state.negative_prompt is not None:
...
```

Note `ComponentSpec` has no notion of an optional component today — declaring it means the block expects it — so part of this is deciding whether we formalize that, or keep the wan_animate_2 shape (don't declare it; consult `requires_unconditional_embeds`, which is `False` when no guider exists).

**State of the codebase**, so we know the size of the pass:

- declares a `guider` spec, reads `components.requires_unconditional_embeds` — `ernie_image`, `flux2`, `helios`, `hunyuan_video1_5`, `krea2`, `ltx`, `qwenimage`, `wan`, `z_image`
- declares a `guider` spec, reads it directly (`components.guider.num_conditions > 1`) — `anima`, `stable_diffusion_xl`
- no guider in the text encoder — `wan_animate_2` (by design), `ltx2` and `stable_diffusion_3` (denoise-only)

plus `requires_unconditional_embeds` copy-pasted into ten `ModularPipeline` subclasses with identical bodies (`flux2` adds an `is_distilled` early return). Once settled it goes into `.ai/modular.md` — this came up writing those docs in #14452.

> Opened by Claude (Opus 5) on behalf of @yiyixuxu.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.