kohya-ss / kohya-ss/sd-scripts
torch 2.3.1+cuda12.1 with flux lora train, but got the train error when use single GPU, pls help
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 1.2k
- Avg merge
- 11m
- Merged PRs (30d)
- 2
Description
## Error
When I use 2 GPU to train flux lora, everything is fine, successful training~, but when I use one GPU or start with 2GPU, but use one, it start to have the error bellow,
I tried :
export NCCL_DEBUG=INFO
export CUDA_DEVICE_ORDER="PCI_BUS_ID"
export NCCL_IB_DISABLE=1


### Environment :
lion-pytorch 0.1.2
open-clip-torch 2.20.0
pytorch-lightning 1.9.0
torch 2.3.1+cu121
torchaudio 2.3.1+cu121
torchmetrics 1.4.1
torchvision 0.18.1+cu121
nvidia-nccl-cu12 2.20.5
centos 8 system


please help!
Contributor guide
No contributing guide indexed for this repository
Research direction
No source files, tests, entry points, or textual traceback are identified; the error details are only in attached images. Start by extracting the traceback and reproducing the Flux LoRA training run with one GPU, then compare it with the successful two-GPU run; done means single-GPU training completes successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100