modelscope / modelscope/DiffSynth-Studio
inquiry for details of wan2.2 i2v lora training?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13.1k
- Forks
- 1.3k
- Avg merge
- 13h 12m
- Merged PRs (30d)
- 45
Description
Hi, DiffSynth-Studio team. I am recently playing with wan2.2 i2v training and found that the token replace style i2v training is newly adopted in wan2.2.
The input_latents' is first initialized with a random noise and the first frame of it is replaced by the input image's latent here.
https://github.com/modelscope/DiffSynth-Studio/blob/fa36739f01bac495eaabeb7d1df27f69b4f5a0d9/diffsynth/pipelines/wan_video_new.py#L658
but later when computing training loss, the first frame is replaced with the noised version of the target video' latent here.
https://github.com/modelscope/DiffSynth-Studio/blob/fa36739f01bac495eaabeb7d1df27f69b4f5a0d9/diffsynth/pipelines/wan_video_new.py#L81
which seems to be different from the inference pipeline of the original wan2.2 pipeline where the first frame is alway replaced with the clean version of the input image's latent.
https://github.com/modelscope/DiffSynth-Studio/blob/fa36739f01bac495eaabeb7d1df27f69b4f5a0d9/diffsynth/pipelines/wan_video_new.py#L461C17-L461C36
I also checked HunyuanI2V's token replace setting and found that there seems to be difference between HuanyuanI2V and wan2.2 i2v.
In HunyuanI2V, the training and inference pipeline are the same and the first frame's latent is alway a clean version of the input image.
https://github.com/Tencent-Hunyuan/HunyuanVideo-I2V/blob/1481c1d5ae88e9905f54f2a3c6a1b68ef2a10528/hyvideo/diffusion/flow/transport.py#L189
And loss function is only performed on frames other than the first frame.
So I am wondering whether the discrepancy between training and inference code is carefully designed in wan2.2 or there might be a mistake in the training code?
Looking forward to your reply. Thanks!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with diffsynth/pipelines/wan_video_new.py at the referenced lines around 81, 461, and 658, then compare the training and inference latent replacement paths with HunyuanVideo-I2V's transport.py around line 189. Trace whether the first-frame treatment is intentional or inconsistent; done means a maintainer-confirmed explanation and, if it is an error, a focused correction with supporting validation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100