kohya-ss / kohya-ss/sd-scripts
NaN showing when training LORA model
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 1.2k
- Avg merge
- 11m
- Merged PRs (30d)
- 2
Description
It kept stopping at caching latents, and I've uninstalled and reset my PC, but the same problem occurred.
accelerator device: cuda
INFO loading model for process 0/1 train_util.py:4385
INFO load StableDiffusion checkpoint: train_util.py:4341
F:/Kohya/kohya_ss/models/SD-v1.5-pruned.ckpt
2024-04-19 03:02:51 INFO UNet2DConditionModel: 64, 8, 768, False, False original_unet.py:1387
2024-04-19 03:02:56 INFO loading u-net: model_util.py:1009
INFO loading vae: model_util.py:1017
2024-04-19 03:02:58 INFO loading text encoder: model_util.py:1074
INFO Enable xformers for U-Net train_util.py:2660
import network module: networks.lora
2024-04-19 03:02:59 INFO [Dataset 0] train_util.py:2079
INFO caching latents. train_util.py:974
INFO checking cache validity... train_util.py:984
100%|██████████████████████████████████████████████████████████████████████████████████████████| 18/18 [00:00
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at library/train_util.py:2428, where cache_batch_latents raises the reported error, and trace the call from train_network.py:272 through dataset.cache_latents. Reproduce with the supplied configuration and the named JPEG, then verify that latent caching completes without the NaN error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100