modelscope / modelscope/DiffSynth-Studio

When I trained the Wan-VACE-14B model with LoRA, an OOM occurred.

Open
#683 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
13.1k
Forks
1.3k
Avg merge
13h 12m
Merged PRs (30d)
45

Description

An OOM error occurred when I used the official LoRA fine-tuning code to train VACE-14B. My code script is as follows:

accelerate launch examples/wanvideo/model_training/train.py \
  --dataset_base_path xxx(my datasets root path) \
  --dataset_metadata_path xxx(my datasets path) \
  --data_file_keys "video,vace_video,vace_reference_image" \
  --height 480 \
  --width 832 \
  --num_frames 81\
  --model_id_with_origin_paths "Wan-AI/Wan2.1-VACE-14B:diffusion_pytorch_model*.safetensors,Wan-AI/Wan2.1-VACE-14B:models_t5_umt5-xxl-enc-bf16.pth,Wan-AI/Wan2.1-VACE-14B:Wan2.1_VAE.pth" \
  --learning_rate 1e-4 \
  --num_epochs 200 \
  --remove_prefix_in_ckpt "pipe.vace." \
  --output_path "./models/train/Wan2.1-VACE-14B_lora" \
  --lora_base_model "vace" \
  --lora_target_modules "q,k,v,o,ffn.0,ffn.2" \
  --lora_rank 16\
  --extra_inputs "vace_video,vace_reference_image" \
  --use_gradient_checkpointing_offload

When I was testing, I found that num_frames could only reach a maximum of 25 under 8 A100 frames. An OOM error will occur if it exceeds 25. How can I solve this problem? Otherwise, only the first 25 frames of the video can be input. If frame extraction training is conducted, the problem of camera shake will occur. (我在测试的时候发现,num_frames在8张A100下面最多只能到25。超过25就会发生OOM错误。我该如何解决这个问题?否则只能输入视频的前25帧,如果抽帧训练会出现镜头抖动的问题。)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with examples/wanvideo/model_training/train.py and the provided accelerate launch command, then compare memory behavior as num_frames increases beyond 25 with 8 A100 GPUs. Done means identifying a reproducible cause or configuration change that permits longer VACE-14B LoRA training without OOM, while preserving the stated video inputs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.