kohya-ss / kohya-ss/sd-scripts

Error on training with DAdaptSGD

Open
#570 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

[Dataset 0]
caching latents.
100%|██████████████████████████████████████████████████████████████████████████████████| 29/29 [00:04<00:00, 6.48it/s]
create LoRA network. base dim (rank): 128, alpha: 64.0
neuron dropout: p=None, rank dropout: p=None, module dropout: p=None
create LoRA for Text Encoder: 72 modules.
create LoRA for U-Net: 192 modules.
enable LoRA for text encoder
enable LoRA for U-Net
preparing optimizer, data loader etc.
use D-Adaptation SGD optimizer | {}
running training / 学習開始
num train images * repeats / 学習画像の数×繰り返し回数: 870
num reg images / 正則化画像の数: 0
num batches per epoch / 1epochのバッチ数: 435
num epochs / epoch数: 10
batch size per device / バッチサイズ: 2
gradient accumulation steps / 勾配を合計するステップ数 = 1
total optimization steps / 学習ステップ数: 4350
steps: 0%| | 0/4350 [00:00

Contributor guide

No contributing guide indexed for this repository

Research direction

The report contains a training log for DAdaptSGD with loss becoming NaN during epoch 2, but names no source file or test. Start by reproducing the reported training run and trace the first non-finite loss through the optimizer and training entry point. Done means identifying the cause, preventing the NaN loss, and confirming that training completes with finite losses.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.