modelscope / modelscope/DiffSynth-Studio
Is it possible to fine-tune Minimax H3 with silent videos?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13.1k
- Forks
- 1.3k
- Avg merge
- 13h 12m
- Merged PRs (30d)
- 45
Description
Here is my cmd:
accelerate launch examples/minimax_h3/model_training/train.py \
--dataset_base_path data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-FL2VA \
--dataset_metadata_path data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-FL2VA/metadata.csv \
--data_file_keys "video" \
--extra_inputs "input_image" \
--height 480 \
--width 832 \
--num_frames 124 \
--dataset_repeat 1 \
--model_id_with_origin_paths "MiniMax/MiniMax-H3:FL2VA/text_encoder/model*.safetensors,MiniMax/MiniMax-H3:FL2VA/video_vae/source/model.safetensors,MiniMax/MiniMax-H3:FL2VA/audio_vae/model.safetensors" \
--learning_rate 1e-4 \
--num_epochs 1 \
--remove_prefix_in_ckpt "pipe.dit." \
--output_path "./models/train/MiniMax-H3-FL2VA-split-cache" \
--lora_base_model "dit" \
--lora_target_modules "qkv_proj,out_proj" \
--lora_rank 32 \
--use_gradient_checkpointing \
--task "sft:data_process"
# FL2VA - stage 2 (train)
accelerate launch examples/minimax_h3/model_training/train.py \
--dataset_base_path ./models/train/MiniMax-H3-FL2VA-split-cache \
--data_file_keys "video" \
--extra_inputs "input_image" \
--height 480 \
--width 832 \
--num_frames 124 \
--dataset_repeat 100 \
--model_id_with_origin_paths "MiniMax/MiniMax-H3:FL2VA/transformer/model*.safetensors" \
--learning_rate 1e-4 \
--num_epochs 5 \
--remove_prefix_in_ckpt "pipe.dit." \
--output_path "./models/train/MiniMax-H3-FL2VA-split" \
--lora_base_model "dit" \
--lora_target_modules "qkv_proj,out_proj" \
--lora_rank 32 \
--use_gradient_checkpointing \
--find_unused_parameters \
--task "sft:train"
I remove input_audio and get this error:
File "/home/test/project/DiffSynth-Studio/diffsynth/diffusion/loss.py", line 78, in FlowMatchSFTMiniMaxH3AudioVideoLoss
audio_noise = torch.randn_like(inputs["audio_input_latents"])
~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'audio_input_latents'
How to fine-tune Minimax H3 using Videos without audio?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with examples/minimax_h3/model_training/train.py and trace the data inputs into diffsynth/diffusion/loss.py, especially FlowMatchSFTMiniMaxH3AudioVideoLoss. Reproduce the command without input_audio and inspect where audio_input_latents is expected. Done means silent-video fine-tuning no longer raises the reported KeyError and the training path has a defined outcome for missing audio.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- audio-video-rtc, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100