modelscope / modelscope/DiffSynth-Studio

inquiry for details of wan2.2 i2v lora training?

Open
#793 2 comments 9 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
13.1k
Forks
1.3k
Avg merge
13h 12m
Merged PRs (30d)
45

Description

Hi, DiffSynth-Studio team. I am recently playing with wan2.2 i2v training and found that the token replace style i2v training is newly adopted in wan2.2.
The input_latents' is first initialized with a random noise and the first frame of it is replaced by the input image's latent here.
https://github.com/modelscope/DiffSynth-Studio/blob/fa36739f01bac495eaabeb7d1df27f69b4f5a0d9/diffsynth/pipelines/wan_video_new.py#L658

but later when computing training loss, the first frame is replaced with the noised version of the target video' latent here.
https://github.com/modelscope/DiffSynth-Studio/blob/fa36739f01bac495eaabeb7d1df27f69b4f5a0d9/diffsynth/pipelines/wan_video_new.py#L81

which seems to be different from the inference pipeline of the original wan2.2 pipeline where the first frame is alway replaced with the clean version of the input image's latent.
https://github.com/modelscope/DiffSynth-Studio/blob/fa36739f01bac495eaabeb7d1df27f69b4f5a0d9/diffsynth/pipelines/wan_video_new.py#L461C17-L461C36

I also checked HunyuanI2V's token replace setting and found that there seems to be difference between HuanyuanI2V and wan2.2 i2v.
In HunyuanI2V, the training and inference pipeline are the same and the first frame's latent is alway a clean version of the input image.
https://github.com/Tencent-Hunyuan/HunyuanVideo-I2V/blob/1481c1d5ae88e9905f54f2a3c6a1b68ef2a10528/hyvideo/diffusion/flow/transport.py#L189
And loss function is only performed on frames other than the first frame.

So I am wondering whether the discrepancy between training and inference code is carefully designed in wan2.2 or there might be a mistake in the training code?

Looking forward to your reply. Thanks!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with diffsynth/pipelines/wan_video_new.py at the referenced lines around 81, 461, and 658, then compare the training and inference latent replacement paths with HunyuanVideo-I2V's transport.py around line 189. Trace whether the first-frame treatment is intentional or inconsistent; done means a maintainer-confirmed explanation and, if it is an error, a focused correction with supporting validation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.