modelscope / modelscope/DiffSynth-Studio

Wan2.1-Fun-1.3B-InP reports an error when fine-tuning LoRa: Channel mismatch

Open
#745 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
13.1k
Forks
1.3k
Avg merge
13h 12m
Merged PRs (30d)
45

Description

Wan2.1-Fun-1.3B-InP reports an error when fine-tuning LoRa: Channel mismatch,The code and error are as follows:
accelerate launch examples/wanvideo/model_training/train.py
--dataset_base_path DiffSynth-Studio/dataset
--dataset_metadata_path DiffSynth-Studio/dataset/metadata.csv
--height 832
--width 480
--dataset_repeat 100
--model_id_with_origin_paths "PAI/Wan2.1-Fun-1.3B-InP:diffusion_pytorch_model*.safetensors,PAI/Wan2.1-Fun-1.3B-InP:models_t5_umt5-xxl-enc-bf16.pth,PAI/Wan2.1-Fun-1.3B-InP:Wan2.1_VAE.pth,PAI/Wan2.1-Fun-1.3B-InP:models_clip_open-clip-xlm-roberta-large-vit-huge-14.pth"
--learning_rate 1e-4
--num_epochs 50
--remove_prefix_in_ckpt "pipe.dit."
--output_path "/home/putao/www/DiffSynth-Studio/models"
--lora_base_model "dit"
--lora_target_modules "q,k,v,o,ffn.0,ffn.2"
--lora_rank 32
--use_gradient_checkpointing_offload

The width and height of my video dataset are already 480*832, in vertical format, but I keep getting the following error:

[rank1]: RuntimeError: Given groups=1, weight of size [1536, 36, 1, 2, 2], expected input[1, 16, 21, 60, 104] to have 36 channels, but got 16 channels instead
[rank0]:[W805 18:07:31.715284739 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator())
E0805 18:07:32.537000 3314691 site-packages/torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: 1) local_rank: 0 (pid: 3314808) of binary: /home/putao/anaconda3/envs/vace/bin/python3.10

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at examples/wanvideo/model_training/train.py and reproduce the reported accelerate launch command with the Wan2.1-Fun-1.3B-InP checkpoint and dataset settings. Trace the model input and checkpoint channel expectations, then verify that fine-tuning completes without the reported mismatch.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.