modelscope / modelscope/DiffSynth-Studio
LoRA trained on Wan2.1-I2V-14B-480P does not work
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13.1k
- Forks
- 1.3k
- Avg merge
- 13h 12m
- Merged PRs (30d)
- 45
Description
Version: 1.1.8
I am encountering an issue where a LoRA trained on the Wan2.1-I2V-14B-480P (Image-to-Video) base model does not produce the expected video output.
Successful Scenario (Working)
- A LoRA was successfully trained using the Wan2.1-T2V-14B (Text-to-Video) base model.
- The video generated using the T2V-trained LoRA exhibits the desired effect.
- Crucially: This T2V-trained LoRA also works correctly when loaded onto the Wan2.1-I2V-14B-480P base model.
Problem Scenario (Not Working)
- A LoRA was trained directly using the Wan2.1-I2V-14B-480P base model.
- When this I2V-trained LoRA is applied, the generated video does not exhibit the desired effect.
Training and Data Details
- Training Commands: I used the exact same command line given in the WanVideo examples section for both the T2V and I2V training runs.
- Dataset: The dataset is identical for both training attempts. It consists of an input image and an associated prompt with just the trigger keyword.
- Metadata Example: I am deliberately omitting a specific descriptive prompt in the
metadata.csvto focus on the style keyword:
video,prompt
training_image_1.png,"N0Y1V2R3S"
training_image_2.png,"N0Y1V2R3S"
...
training_image_250.png,"N0Y1V2R3S"
ComfyUI Usage (Context, Not the Issue)
For testing, I used ComfyUI, ensuring I loaded the LoRA via the WanVideo-specific nodes. (Note: Standard ComfyUI LoRA loading nodes fail due to key name differences in the safetensor file produced by DiffSynth-Studio, but I have verified this is not the root cause of the I2V training failure.)
Summary of the Problem
The core issue is that I2V-native LoRA training is failing to capture the desired style/effect, even though the same dataset, training command, and environment allow for successful T2V LoRA training that can then be applied to the I2V model.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the WanVideo example training command and the metadata.csv dataset, then reproduce the identical T2V and I2V runs using the reported version 1.1.8 setup. Compare the resulting LoRA behavior through ComfyUI's WanVideo-specific nodes. Done means an LoRA trained on Wan2.1-I2V-14B-480P captures the intended effect when applied to that base model.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100