modelscope / modelscope/DiffSynth-Studio
BrokenPipeError: [Errno 32] Broken pipe
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13.1k
- Forks
- 1.3k
- Avg merge
- 13h 12m
- Merged PRs (30d)
- 45
Description
我们用4张32G运行train_flux_lora.py训练FLUX lora时报错:
[2025-06-17 20:04:09,900] [INFO] [real_accelerator.py:254:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-06-17 20:04:10,569] [INFO] [real_accelerator.py:254:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-06-17 20:04:11,024] [INFO] [real_accelerator.py:254:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-06-17 20:04:11,812] [INFO] [logging.py:107:log_dist] [Rank -1] [TorchCheckpointEngine] Initialized with serialization = False
initializing deepspeed distributed: GLOBAL_RANK: 2, MEMBER: 3/4
[2025-06-17 20:04:12,101] [INFO] [logging.py:107:log_dist] [Rank -1] [TorchCheckpointEngine] Initialized with serialization = False
initializing deepspeed distributed: GLOBAL_RANK: 3, MEMBER: 4/4
[2025-06-17 20:04:12,494] [INFO] [logging.py:107:log_dist] [Rank -1] [TorchCheckpointEngine] Initialized with serialization = False
initializing deepspeed distributed: GLOBAL_RANK: 1, MEMBER: 2/4
Exception ignored in: <_io.BufferedWriter name=54>
BrokenPipeError: [Errno 32] Broken pipe
Segmentation fault (core dumped)
参数:CUDA_VISIBLE_DEVICES="0,1,2,3" python examples/train/flux/train_flux_lora.py
--pretrained_text_encoder_path models/FLUX/FLUX.1-dev/text_encoder/model.safetensors
--pretrained_text_encoder_2_path models/FLUX/FLUX.1-dev/text_encoder_2
--pretrained_dit_path models/FLUX/FLUX.1-dev/flux1-dev.safetensors
--pretrained_vae_path models/FLUX/FLUX.1-dev/ae.safetensors
--dataset_path data/cel-shading
--output_path ./models
--max_epochs 1
--steps_per_epoch 100
--height 1024
--width 1024
--center_crop
--precision "bf16"
--learning_rate 1e-4
--lora_rank 16
--lora_alpha 16
--use_gradient_checkpointing
--align_to_opensource_format
--training_strategy deepspeed_stage_2
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with examples/train/flux/train_flux_lora.py and reproduce the command using CUDA_VISIBLE_DEVICES across four GPUs. Inspect the DeepSpeed distributed initialization and TorchCheckpointEngine messages around the BrokenPipeError and segmentation fault; done means multi-GPU training starts without either failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100