huggingface / huggingface/diffusers

Wan I2V expand_timesteps: why hard-clamp only first frame (no last-frame clamp)?

Offen
#13,167 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
34.5k
Forks
7.3k
Ø Merge
3 T. 3 Std.
Gemergte PRs (30 T.)
91

Beschreibung

I have a question on `expand_timesteps` option within `WanImageToVideoPipeline`. I see that the mechanism hard-clamps only the first frame. For FLF (first+last) conditioning, is there a reason we can’t (or shouldn’t) apply the same mechanism to *both* endpoints?

The related code is in:
- `diffusers/src/diffusers/pipelines/wan/pipeline_wan_i2v.py`

In `prepare_latents`, when `expand_timesteps` is enabled, `video_condition` ignores `last_image` and `first_frame_mask` is defined only for frame 0:
```py
# prepare_latents
if self.config.expand_timesteps:
video_condition = image

...
if self.config.expand_timesteps:
first_frame_mask = torch.ones(1, 1, num_latent_frames, latent_height, latent_width, ...)
first_frame_mask[:, :, 0] = 0
return latents, latent_condition, first_frame_mask
```
In the denoising loop, the clamp/mix uses only first_frame_mask:

```py

if self.config.expand_timesteps:
latent_model_input = (1 - first_frame_mask) * condition + first_frame_mask * latents
temp_ts = (first_frame_mask[0][0][:, ::2, ::2] * t).flatten()
```

Question
Is it a limitation of the Wan2.2 I2V checkpoint training (only first-frame conditioning)?
Or a design choice because per-token timestep masking doesn’t extend cleanly to a last-frame constraint?

I see that the corresponding part was written by @yiyixuxu, thanks!

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Start in diffusers/src/diffusers/pipelines/wan/pipeline_wan_i2v.py and trace prepare_latents, first_frame_mask, and the denoising-loop clamp/mix. Check whether Wan2.2 I2V training supports last-frame conditioning or whether timestep masking prevents it; done means documenting the reason or defining the required behavior for both endpoints.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python, pytorch
Bereich
machine-learning
Issue-Typ
Feature
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Muss geklärt werden
Anfängerfreundlichkeit
35/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.