kohya-ss / kohya-ss/sd-scripts

torch 2.3.1+cuda12.1 with flux lora train, but got the train error when use single GPU, pls help

Open
#1,554 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

## Error
When I use 2 GPU to train flux lora, everything is fine, successful training~, but when I use one GPU or start with 2GPU, but use one, it start to have the error bellow,
I tried :
export NCCL_DEBUG=INFO
export CUDA_DEVICE_ORDER="PCI_BUS_ID"
export NCCL_IB_DISABLE=1
![image](https://github.com/user-attachments/assets/93ec1b76-f027-4692-b5ad-04f11772f22f)
![image](https://github.com/user-attachments/assets/16c9337c-f980-43d8-8ea0-7d94bf1137e1)
### Environment :
lion-pytorch 0.1.2
open-clip-torch 2.20.0
pytorch-lightning 1.9.0
torch 2.3.1+cu121
torchaudio 2.3.1+cu121
torchmetrics 1.4.1
torchvision 0.18.1+cu121
nvidia-nccl-cu12 2.20.5
centos 8 system
![image](https://github.com/user-attachments/assets/6eab30ff-4bb9-402e-81fb-10fc891f2b2c)
![image](https://github.com/user-attachments/assets/655ba487-9d91-43be-9250-7fccf4d3b08c)
please help!

Contributor guide

No contributing guide indexed for this repository

Research direction

No source files, tests, entry points, or textual traceback are identified; the error details are only in attached images. Start by extracting the traceback and reproducing the Flux LoRA training run with one GPU, then compare it with the successful two-GPU run; done means single-GPU training completes successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.