aigc-apps / aigc-apps/VideoX-Fun

wan的视频生成训练后,推理生成的视频有很强的抖动感

Open
#475 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.3k
Forks
188
Avg merge
2d 22h
Merged PRs (30d)
3

Description

您好,我用该仓库下的code进行了模型训练,首先我尝试了wan2.1-i2v-14v-480p的训练,以下是我的训练超参数:
export MASTER_ADDR="10.86.17.140"
export MASTER_PORT=29500
export WORLD_SIZE=3 # The number of machines
export NUM_PROCESS=24 # The number of processes, such as WORLD_SIZE * 8
export RANK=0 # The rank of this machine
export MODEL_NAME="/mmu_mllm_hdd_2/zuofei/model_param/Wan2.1-I2V-14B-480P"
#export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="/mmu_mllm_hdd_2/zuofei/VideoX-Fun/datasets/i2v_traing.json"

NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.

export NCCL_IB_DISABLE=1

export NCCL_P2P_DISABLE=1

NCCL_DEBUG=INFO

accelerate launch --main_process_ip=$MASTER_ADDR --main_process_port=$MASTER_PORT --num_machines=$WORLD_SIZE --num_processes=$NUM_PROCESS --machine_rank=$RANK --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TRANSFORMER_BASED_WRAP --fsdp_transformer_layer_cls_to_wrap=WanAttentionBlock --fsdp_sharding_strategy "FULL_SHARD" --fsdp_state_dict_type=SHARDED_STATE_DICT --fsdp_backward_prefetch "BACKWARD_PRE" --fsdp_cpu_ram_efficient_loading False scripts/wan2.1/train.py
--config_path="config/wan2.1/wan_civitai.yaml"
--pretrained_model_name_or_path=$MODEL_NAME
--train_data_dir=$DATASET_NAME
--train_data_meta=$DATASET_META_NAME
--image_sample_size=640
--video_sample_size=640
--token_sample_size=640
--video_sample_stride=2
--video_sample_n_frames=121
--train_batch_size=1
--video_repeat=1
--gradient_accumulation_steps=2
--dataloader_num_workers=8
--num_train_epochs=3
--checkpointing_steps=200
--learning_rate=2e-05
--lr_scheduler="constant_with_warmup"
--lr_warmup_steps=100
--seed=42
--output_dir="2.1_i2v_output_dir"
--gradient_checkpointing
--mixed_precision="bf16"
--adam_weight_decay=3e-2
--adam_epsilon=1e-10
--vae_mini_batch=1
--max_grad_norm=0.05
--random_hw_adapt
--training_with_video_token_length
--enable_bucket
--uniform_sampling
--low_vram
--train_mode="i2v"
--trainable_modules "."

我发现训练后的视频有很强的抖动感,明显特别不稳定:
训练前:
https://github.com/user-attachments/assets/a276b808-5c01-446a-a090-d57c8f83b1a6
训练后:
https://github.com/user-attachments/assets/b44b3b34-28f4-4ecf-b1a0-799142efd3d0
想问一下这是什么问题导致的呢?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with scripts/wan2.1/train.py and config/wan2.1/wan_civitai.yaml, then review the supplied training command and hyperparameters. Reproduce the i2v training and inference setup if the referenced videos remain available, comparing the pre-training and post-training outputs. Done means identifying the source of the jitter and documenting a verified corrective change or configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.