kohya-ss / kohya-ss/sd-scripts

NaN showing when training LORA model

Open
#1,283 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

It kept stopping at caching latents, and I've uninstalled and reset my PC, but the same problem occurred.

accelerator device: cuda
INFO loading model for process 0/1 train_util.py:4385
INFO load StableDiffusion checkpoint: train_util.py:4341
F:/Kohya/kohya_ss/models/SD-v1.5-pruned.ckpt
2024-04-19 03:02:51 INFO UNet2DConditionModel: 64, 8, 768, False, False original_unet.py:1387
2024-04-19 03:02:56 INFO loading u-net: model_util.py:1009
INFO loading vae: model_util.py:1017
2024-04-19 03:02:58 INFO loading text encoder: model_util.py:1074
INFO Enable xformers for U-Net train_util.py:2660
import network module: networks.lora
2024-04-19 03:02:59 INFO [Dataset 0] train_util.py:2079
INFO caching latents. train_util.py:974
INFO checking cache validity... train_util.py:984
100%|██████████████████████████████████████████████████████████████████████████████████████████| 18/18 [00:00

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at library/train_util.py:2428, where cache_batch_latents raises the reported error, and trace the call from train_network.py:272 through dataset.cache_latents. Reproduce with the supplied configuration and the named JPEG, then verify that latent caching completes without the NaN error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.