modelscope / modelscope/DiffSynth-Studio
The bug for wan2.1 I2V 14B lora training. Data processing.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13.1k
- Forks
- 1.3k
- Avg merge
- 13h 12m
- Merged PRs (30d)
- 45
Description
I found that if you prepare the data using this command
" CUDA_VISIBLE_DEVICES="0,1" python examples/wanvideo/train_wan_t2v.py
--task train
--train_architecture lora
--dataset_path data/example_dataset
--output_path ./models
--dit_path "/path to model/Wan2.1-I2V-14B-480P/diffusion_pytorch_model-00001-of-00007.safetensors,
/path to modelWan2.1-I2V-14B-480P/diffusion_pytorch_model-00002-of-00007.safetensors,
/path to model/Wan2.1-I2V-14B-480P/diffusion_pytorch_model-00003-of-00007.safetensors,
/path to modelWan2.1-I2V-14B-480P/diffusion_pytorch_model-00004-of-00007.safetensors,
/path to modelWan2.1-I2V-14B-480P/diffusion_pytorch_model-00005-of-00007.safetensors,
/path to model/Wan2.1-I2V-14B-480P/diffusion_pytorch_model-00006-of-00007.safetensors,
/path to modelWan2.1-I2V-14B-480P/diffusion_pytorch_model-00007-of-00007.safetensors"
--steps_per_epoch 500
--max_epochs 10 \
--learning_rate 1e-4 \
--lora_rank 16
--lora_alpha 16
--lora_target_modules "q,k,v,o,ffn.0,ffn.2"
--accumulate_grad_batches 1
--use_gradient_checkpointing
--training_strategy ddp
--image_encoder_path "/path to model/Wan2.1-I2V-14B-480P/models_clip_open-clip-xlm-roberta-large-vit-huge-14.pth" "
You need to change the code train_wan_t2v.py,
if "clip_feature" in image_emb:
image_emb["clip_feature"] = image_emb["clip_feature"][0].to(self.device)
"if "clip_fea" in image_emb:
image_emb["clip_fea"] = image_emb["clip_fea"][0].to(self.device)".
That is, the key in image_emb (video_00001.mp4.tensors.pth, generated by CLIP) is "clip_fea" (not "clip_feature"),
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in examples/wanvideo/train_wan_t2v.py and inspect how image_emb keys are handled during data processing. Compare the generated video_00001.mp4.tensors.pth contents with the command's Wan2.1 I2V training path, focusing on the reported clip_fea key. Done means the training path accepts the generated tensors without requiring clip_feature, then the supplied command can proceed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 50/100