kohya-ss / kohya-ss/sd-scripts
CUDA detection failed
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 1.2k
- Avg merge
- 11m
- Merged PRs (30d)
- 2
Description
Hello, I'm getting some errors when attempting to start training LoRA, and I'm not sure how I could go about debugging this. I was hoping someone could offer me some directions. I was able to get training started, but after the training images are cached I'm getting an error message saying it can't detect a CUDA driver. Now, I'm using the Kohya GUI on a mobile 3060 with 6GB of VRAM, and I use both A1111 WebUI and Easy Diffusion frequently so I know CUDA drivers are working, but I'm not sure why Kohya is having trouble finding them.
Below is relevant error log I'm getting. Any assistance in fixing this would be very appreciated.
```
CUDA SETUP: TODO: compile library for specific version: libbitsandbytes_cuda116.dll
CUDA SETUP: Defaulting to libbitsandbytes.so...
CUDA SETUP: CUDA detection failed. Either CUDA driver not installed, CUDA not installed, or you have multiple conflicting CUDA libraries!
CUDA SETUP: If you compiled from source, try again with `make CUDA_VERSION=DETECTED_CUDA_VERSION` for example, `make CUDA_VERSION=113`.
Traceback (most recent call last):
File "C:\Kohya\kohya_ss\train_db.py", line 346, in
train(args)
File "C:\Kohya\kohya_ss\train_db.py", line 122, in train
import bitsandbytes as bnb
File "C:\Kohya\kohya_ss\venv\lib\site-packages\bitsandbytes\__init__.py", line 6, in
from .autograd._functions import (
File "C:\Kohya\kohya_ss\venv\lib\site-packages\bitsandbytes\autograd\_functions.py", line 5, in
import bitsandbytes.functional as F
File "C:\Kohya\kohya_ss\venv\lib\site-packages\bitsandbytes\functional.py", line 13, in
from .cextension import COMPILED_WITH_CUDA, lib
File "C:\Kohya\kohya_ss\venv\lib\site-packages\bitsandbytes\cextension.py", line 43, in
lib = CUDALibrary_Singleton.get_instance().lib
File "C:\Kohya\kohya_ss\venv\lib\site-packages\bitsandbytes\cextension.py", line 39, in get_instance
cls._instance.initialize()
File "C:\Kohya\kohya_ss\venv\lib\site-packages\bitsandbytes\cextension.py", line 27, in initialize
raise Exception('CUDA SETUP: Setup Failed!')
Exception: CUDA SETUP: Setup Failed!
Traceback (most recent call last):
File "C:\Users\andre\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\andre\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Kohya\kohya_ss\venv\Scripts\accelerate.exe\__main__.py", line 7, in
File "C:\Kohya\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 45, in main
args.func(args)
File "C:\Kohya\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1104, in launch_command
simple_launcher(args)
File "C:\Kohya\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 567, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['C:\\Kohya\\kohya_ss\\venv\\Scripts\\python.exe', 'train_db.py', '--pretrained_model_name_or_path=C:\\sd-shared-files\\models\\sd-v1-5-pruned-emaonly.ckpt', '--train_data_dir=C:/Temp/omgcsply lora/img', '--resolution=512,512', '--output_dir=C:/Temp/omgcsply lora/model', '--logging_dir=C:/Temp/omgcsply lora/log', '--save_model_as=safetensors', '--max_data_loader_n_workers=1', '--learning_rate=0.0001', '--lr_scheduler=constant', '--train_batch_size=1', '--max_train_steps=6800', '--save_every_n_epochs=1', '--mixed_precision=fp16', '--save_precision=fp16', '--seed=1234', '--caption_extension=.txt', '--cache_latents', '--max_data_loader_n_workers=1', '--clip_skip=2', '--bucket_reso_steps=64', '--mem_eff_attn', '--gradient_checkpointing', '--xformers', '--use_8bit_adam', '--bucket_no_upscale']' returned non-zero exit status 1.
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with train_db.py at the reported import of bitsandbytes, then inspect the installed bitsandbytes files under venv\lib\site-packages and the full accelerate command in the log. Reproduce the failure with the shown training invocation and determine whether the CUDA detection error is specific to this environment. Done means identifying a reproducible cause and documenting a verified resolution or the missing project-side change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100