modelscope / modelscope/DiffSynth-Studio
Qwen-image的llora微调支持拆分+Fp8吗?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13.1k
- Forks
- 1.3k
- Avg merge
- 13h 12m
- Merged PRs (30d)
- 45
Description
我把拆分训练的第二阶段代码写成了下面的形式,请问这个可以起到FP8冻结底座的效果吗?
谢谢!
accelerate launch --num_processes 4 ./DiffSynth-Studio/examples/qwen_image/model_training/train.py
--dataset_base_path "./models/train/Qwen-Image-LoRA-splited-cache"
--max_pixels 1048576
--dataset_repeat 50
--model_paths '[
[
"/data/yhwang/qwenimage/Qwen/Qwen-Image/transformer/diffusion_pytorch_model-00001-of-00009.safetensors",
"/data/yhwang/qwenimage/Qwen/Qwen-Image/transformer/diffusion_pytorch_model-00002-of-00009.safetensors",
"/data/yhwang/qwenimage/Qwen/Qwen-Image/transformer/diffusion_pytorch_model-00003-of-00009.safetensors",
"/data/yhwang/qwenimage/Qwen/Qwen-Image/transformer/diffusion_pytorch_model-00004-of-00009.safetensors",
"/data/yhwang/qwenimage/Qwen/Qwen-Image/transformer/diffusion_pytorch_model-00005-of-00009.safetensors",
"/data/yhwang/qwenimage/Qwen/Qwen-Image/transformer/diffusion_pytorch_model-00006-of-00009.safetensors",
"/data/yhwang/qwenimage/Qwen/Qwen-Image/transformer/diffusion_pytorch_model-00007-of-00009.safetensors",
"/data/yhwang/qwenimage/Qwen/Qwen-Image/transformer/diffusion_pytorch_model-00008-of-00009.safetensors",
"/data/yhwang/qwenimage/Qwen/Qwen-Image/transformer/diffusion_pytorch_model-00009-of-00009.safetensors"
]
]'
--learning_rate 1e-4
--num_epochs 5
--remove_prefix_in_ckpt "pipe.dit."
--output_path "./models/train/Qwen-Image-LoRA-splited"
--lora_base_model "dit"
--lora_target_modules "to_q,to_k,to_v,add_q_proj,add_k_proj,add_v_proj,to_out.0,to_add_out,img_mlp.net.2,img_mod.1,txt_mlp.net.2,txt_mod.1"
--lora_rank 32
--use_gradient_checkpointing
--use_gradient_checkpointing_offload
--dataset_num_workers 4
--find_unused_parameters
--task "sft:train"
--fp8_models '[
[
"/data/yhwang/qwenimage/Qwen/Qwen-Image/transformer/diffusion_pytorch_model-00001-of-00009.safetensors",
"/data/yhwang/qwenimage/Qwen/Qwen-Image/transformer/diffusion_pytorch_model-00002-of-00009.safetensors",
"/data/yhwang/qwenimage/Qwen/Qwen-Image/transformer/diffusion_pytorch_model-00003-of-00009.safetensors",
"/data/yhwang/qwenimage/Qwen/Qwen-Image/transformer/diffusion_pytorch_model-00004-of-00009.safetensors",
"/data/yhwang/qwenimage/Qwen/Qwen-Image/transformer/diffusion_pytorch_model-00005-of-00009.safetensors",
"/data/yhwang/qwenimage/Qwen/Qwen-Image/transformer/diffusion_pytorch_model-00006-of-00009.safetensors",
"/data/yhwang/qwenimage/Qwen/Qwen-Image/transformer/diffusion_pytorch_model-00007-of-00009.safetensors",
"/data/yhwang/qwenimage/Qwen/Qwen-Image/transformer/diffusion_pytorch_model-00008-of-00009.safetensors",
"/data/yhwang/qwenimage/Qwen/Qwen-Image/transformer/diffusion_pytorch_model-00009-of-00009.safetensors"
],
]'
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with DiffSynth-Studio/examples/qwen_image/model_training/train.py and trace how --model_paths and --fp8_models are parsed and applied during LoRA training. Check the handling of sharded Qwen-Image transformer weights and determine whether the command provides a frozen FP8 base model; done means the supported behavior or required changes are clearly documented and validated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100