modelscope / modelscope/DiffSynth-Studio

Flux.2训练lora时无法加载transformer模型

Open
#1,252 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
13.1k
Forks
1.3k
Avg merge
13h 12m
Merged PRs (30d)
45

Description

我基于Flux.2的FLUX.2-dev.sh的脚本(简单改动)进行lora两阶段拆分训练,提前将模型通过modelscope下载到本地并确定模型完整。现在sft:data_process阶段能够正常结束并生成cache文件,但在第二阶段加载transformer模型时会报错:
ValueError: Cannot detect the model type. File: /root/DiffSynth-Studio/models/flux2/transformer/diffusion_pytorch_model-00001-of-00007.safetensors. Model hash: 706251fd7d252e9b87616f71b5cf306d
我试过加载单一transformer文件依然报错:ValueError: Cannot detect the model type. File: /root/DiffSynth-Studio/models/flux2/flux2-dev.safetensors. Model hash: 76bded5d9167873fba768a0ed3e52d3d

我确定模型文件没有问题(通过modelscope下载,并通过hash校验),感觉像是bug。

git信息
commit b4f6a4de6c906763bb68ae074302217c081e872a (HEAD, origin/main, origin/HEAD)

FLUX.2-dev.sh配置
accelerate launch ../train.py
--max_pixels 1048576
--model_paths '[
[
"/root/DiffSynth-Studio/models/flux2/text_encoder/model-00001-of-00010.safetensors",
"/root/DiffSynth-Studio/models/flux2/text_encoder/model-00002-of-00010.safetensors",
"/root/DiffSynth-Studio/models/flux2/text_encoder/model-00003-of-00010.safetensors",
"/root/DiffSynth-Studio/models/flux2/text_encoder/model-00004-of-00010.safetensors",
"/root/DiffSynth-Studio/models/flux2/text_encoder/model-00005-of-00010.safetensors",
"/root/DiffSynth-Studio/models/flux2/text_encoder/model-00006-of-00010.safetensors",
"/root/DiffSynth-Studio/models/flux2/text_encoder/model-00007-of-00010.safetensors",
"/root/DiffSynth-Studio/models/flux2/text_encoder/model-00008-of-00010.safetensors",
"/root/DiffSynth-Studio/models/flux2/text_encoder/model-00009-of-00010.safetensors",
"/root/DiffSynth-Studio/models/flux2/text_encoder/model-00010-of-00010.safetensors"
],
"/root/DiffSynth-Studio/models/flux2/vae/diffusion_pytorch_model.safetensors"
]'
--fp8_models '[
[
"/root/DiffSynth-Studio/models/flux2/text_encoder/model-00001-of-00010.safetensors",
"/root/DiffSynth-Studio/models/flux2/text_encoder/model-00002-of-00010.safetensors",
"/root/DiffSynth-Studio/models/flux2/text_encoder/model-00003-of-00010.safetensors",
"/root/DiffSynth-Studio/models/flux2/text_encoder/model-00004-of-00010.safetensors",
"/root/DiffSynth-Studio/models/flux2/text_encoder/model-00005-of-00010.safetensors",
"/root/DiffSynth-Studio/models/flux2/text_encoder/model-00006-of-00010.safetensors",
"/root/DiffSynth-Studio/models/flux2/text_encoder/model-00007-of-00010.safetensors",
"/root/DiffSynth-Studio/models/flux2/text_encoder/model-00008-of-00010.safetensors",
"/root/DiffSynth-Studio/models/flux2/text_encoder/model-00009-of-00010.safetensors",
"/root/DiffSynth-Studio/models/flux2/text_encoder/model-00010-of-00010.safetensors"
],
"/root/DiffSynth-Studio/models/flux2/vae/diffusion_pytorch_model.safetensors"
]'
--lora_target_modules "to_q,to_k,to_v,add_q_proj,add_k_proj,add_v_proj,to_qkv_mlp_proj,to_out.0,to_add_out,linear_in,linear_out,single_transformer_blocks.0.attn.to_out,single_transformer_blocks.1.attn.to_out,single_transformer_blocks.2.attn.to_out,single_transformer_blocks.3.attn.to_out,single_transformer_blocks.4.attn.to_out,single_transformer_blocks.5.attn.to_out,single_transformer_blocks.6.attn.to_out,single_transformer_blocks.7.attn.to_out,single_transformer_blocks.8.attn.to_out,single_transformer_blocks.9.attn.to_out,single_transformer_blocks.10.attn.to_out,single_transformer_blocks.11.attn.to_out,single_transformer_blocks.12.attn.to_out,single_transformer_blocks.13.attn.to_out,single_transformer_blocks.14.attn.to_out,single_transformer_blocks.15.attn.to_out,single_transformer_blocks.16.attn.to_out,single_transformer_blocks.17.attn.to_out,single_transformer_blocks.18.attn.to_out,single_transformer_blocks.19.attn.to_out,single_transformer_blocks.20.attn.to_out,single_transformer_blocks.21.attn.to_out,single_transformer_blocks.22.attn.to_out,single_transformer_blocks.23.attn.to_out,single_transformer_blocks.24.attn.to_out,single_transformer_blocks.25.attn.to_out,single_transformer_blocks.26.attn.to_out,single_transformer_blocks.27.attn.to_out,single_transformer_blocks.28.attn.to_out,single_transformer_blocks.29.attn.to_out,single_transformer_blocks.30.attn.to_out,single_transformer_blocks.31.attn.to_out,single_transformer_blocks.32.attn.to_out,single_transformer_blocks.33.attn.to_out,single_transformer_blocks.34.attn.to_out,single_transformer_blocks.35.attn.to_out,single_transformer_blocks.36.attn.to_out,single_transformer_blocks.37.attn.to_out,single_transformer_blocks.38.attn.to_out,single_transformer_blocks.39.attn.to_out,single_transformer_blocks.40.attn.to_out,single_transformer_blocks.41.attn.to_out,single_transformer_blocks.42.attn.to_out,single_transformer_blocks.43.attn.to_out,single_transformer_blocks.44.attn.to_out,single_transformer_blocks.45.attn.to_out,single_transformer_blocks.46.attn.to_out,single_transformer_blocks.47.attn.to_out"
--tokenizer_path /root/DiffSynth-Studio/models/flux2/tokenizer
--remove_prefix_in_ckpt "pipe.dit."
--lora_base_model "dit"
--dataset_base_path ./training/20260203/dataset
--dataset_metadata_path ./training/20260203/dataset/metadata.csv
--dataset_repeat 1
--dataset_num_workers 8
--output_path ./training/20260203/output_fp8_r32_epo30/cache
--learning_rate 1e-4
--num_epochs 30
--lora_rank 32
--find_unused_parameters
--use_gradient_checkpointing
--use_gradient_checkpointing_offload
--task "sft:data_process"

accelerate launch ../train.py
--max_pixels 1048576
--model_paths '[
"/root/DiffSynth-Studio/models/flux2/flux2-dev.safetensors"
]'
--fp8_models '[
"/root/DiffSynth-Studio/models/flux2/flux2-dev.safetensors"
]'
--lora_target_modules "to_q,to_k,to_v,add_q_proj,add_k_proj,add_v_proj,to_qkv_mlp_proj,to_out.0,to_add_out,linear_in,linear_out,single_transformer_blocks.0.attn.to_out,single_transformer_blocks.1.attn.to_out,single_transformer_blocks.2.attn.to_out,single_transformer_blocks.3.attn.to_out,single_transformer_blocks.4.attn.to_out,single_transformer_blocks.5.attn.to_out,single_transformer_blocks.6.attn.to_out,single_transformer_blocks.7.attn.to_out,single_transformer_blocks.8.attn.to_out,single_transformer_blocks.9.attn.to_out,single_transformer_blocks.10.attn.to_out,single_transformer_blocks.11.attn.to_out,single_transformer_blocks.12.attn.to_out,single_transformer_blocks.13.attn.to_out,single_transformer_blocks.14.attn.to_out,single_transformer_blocks.15.attn.to_out,single_transformer_blocks.16.attn.to_out,single_transformer_blocks.17.attn.to_out,single_transformer_blocks.18.attn.to_out,single_transformer_blocks.19.attn.to_out,single_transformer_blocks.20.attn.to_out,single_transformer_blocks.21.attn.to_out,single_transformer_blocks.22.attn.to_out,single_transformer_blocks.23.attn.to_out,single_transformer_blocks.24.attn.to_out,single_transformer_blocks.25.attn.to_out,single_transformer_blocks.26.attn.to_out,single_transformer_blocks.27.attn.to_out,single_transformer_blocks.28.attn.to_out,single_transformer_blocks.29.attn.to_out,single_transformer_blocks.30.attn.to_out,single_transformer_blocks.31.attn.to_out,single_transformer_blocks.32.attn.to_out,single_transformer_blocks.33.attn.to_out,single_transformer_blocks.34.attn.to_out,single_transformer_blocks.35.attn.to_out,single_transformer_blocks.36.attn.to_out,single_transformer_blocks.37.attn.to_out,single_transformer_blocks.38.attn.to_out,single_transformer_blocks.39.attn.to_out,single_transformer_blocks.40.attn.to_out,single_transformer_blocks.41.attn.to_out,single_transformer_blocks.42.attn.to_out,single_transformer_blocks.43.attn.to_out,single_transformer_blocks.44.attn.to_out,single_transformer_blocks.45.attn.to_out,single_transformer_blocks.46.attn.to_out,single_transformer_blocks.47.attn.to_out"
--tokenizer_path /root/DiffSynth-Studio/models/flux2/tokenizer
--remove_prefix_in_ckpt "pipe.dit."
--lora_base_model "dit"
--dataset_base_path ./training/20260203/output_fp8_r32_epo30/cache
--dataset_repeat 5
--dataset_num_workers 8
--output_path ./training/20260203/output_fp8_r32_epo30
--learning_rate 1e-4
--num_epochs 30
--lora_rank 32
--find_unused_parameters
--use_gradient_checkpointing
--use_gradient_checkpointing_offload
--task "sft:train"

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the second accelerate launch of ../train.py in FLUX.2-dev.sh, using the reported flux2-dev.safetensors path, and reproduce the model-type detection error. Trace the transformer loading path from train.py and compare it with the first data_process command. Done means the local FLUX.2 transformer loads successfully and the sft:train stage can proceed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.