kohya-ss / kohya-ss/sd-scripts

Multi GPU train of flux report error

Open
#1,475 27 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

I use this setting below to train flux lora:

```
accelerate launch --gpu_ids 0,1 --main_process_port 29502 --mixed_precision bf16 --num_cpu_threads_per_process=2 \
flux_train_network.py --pretrained_model_name_or_path ${flux_model_path} \
--clip_l ${clip_l_path} --t5xxl ${t5xxl_path} --ae ${ae_path} \
--cache_latents_to_disk --save_model_as safetensors --sdpa --persistent_data_loader_workers \
--max_data_loader_n_workers 2 --seed 42 \
--gradient_checkpointing \
--save_precision bf16 --mixed_precision bf16 \
--network_module networks.lora_flux \
--network_dim 16 \
--optimizer_type prodigy \
--learning_rate 1 --network_train_unet_only \
--cache_text_encoder_outputs --cache_text_encoder_outputs_to_disk \
--highvram \
--max_train_epochs 10 \
--save_every_n_epochs 1 \
--train_data_dir=${input_path} \
--output_dir ${output_path} \
--output_name flux_shot \
--timestep_sampling sigmoid --model_prediction_type raw --guidance_scale 1 --loss_type l2 \
--resolution="1024,1024" --bucket_reso_steps=64 --bucket_no_upscale --min_bucket_reso=256 --max_bucket_reso=2048 --enable_bucket \
--caption_extension=".txt" \
--lr_scheduler="cosine" --lr_warmup_steps=396 --train_batch_size=4 --deepspeed --zero_stage=2 \
--log_with="wandb" --wandb_run_name="shot2" --wandb_api_key="" --logging_dir=${output_path}"/logs" --log_tracker_name="flux_lora1"
```
it will report the error like this :
![image](https://github.com/user-attachments/assets/845081fe-d153-43ea-975c-0c65fa7dab18)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the command with accelerate and deepspeed against flux_train_network.py, then inspect the attached error report and the multi-GPU launch configuration. Trace the relevant training path, including networks.lora_flux, to identify why the two-GPU run fails. Done means the supplied command completes multi-GPU training without the reported error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.