kohya-ss / kohya-ss/sd-scripts
[bug] --skip_until_initial_step 使用時、--gradient_accumulation_steps > 1 で再開位置を過剰にスキップして学習が終了する
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 1.2k
- Avg merge
- 11m
- Merged PRs (30d)
- 2
Description
## 事象
--resume で学習 state を再開し、--skip_until_initial_step を指定すると、--gradient_accumulation_steps が 2 以上の場合に、再開済み step 数より大幅に多くの epoch がスキップされます。
その結果、内部の step 数が負値まで減少し、学習が開始されずにすぐ終了します。
## 再現条件
--resume で state を再開
--skip_until_initial_step を指定
--gradient_accumulation_steps=4
再開 state の current_step=500
1 epoch あたりの dataloader batch 数が 100 の例
この条件では、500 optimizer step は 2000 dataloader batch に相当するため、本来は 20 epoch をスキップする必要があります。
しかし実際には 80 epoch がスキップされ、step 数が負値まで進みます。
再開 state は正常に読み込まれますが、--skip_until_initial_step 実行後に epoch のスキップ数が過剰となります。
最終的に学習ループへ入らず終了します。
## ログ
```
running training / 学習開始
num train images * repeats / 学習画像の数×繰り返し回数: 100
num validation images * repeats / 学習画像の数×繰り返し回数: 0
num reg images / 正則化画像の数: 0
num batches per epoch / 1epochのバッチ数: 100
num epochs / epoch数: 30
batch size per device / バッチサイズ: 1
gradient accumulation steps / 勾配を合計するステップ数 = 4
total optimization steps / 学習ステップ数: 750
2026-07-21 07:38:46 INFO skipping 500 steps / 500ステップをスキップします train_network.py:1461
INFO text_encoder is not needed for training. deleting to save memory. train_network.py:1494
2026-07-21 07:38:47 INFO skipping epoch 1 because initial_step (multiplied) is 2000 train_network.py:1514
INFO skipping epoch 2 because initial_step (multiplied) is 1900 train_network.py:1514
INFO skipping epoch 3 because initial_step (multiplied) is 1800 train_network.py:1514
INFO skipping epoch 4 because initial_step (multiplied) is 1700 train_network.py:1514
INFO skipping epoch 5 because initial_step (multiplied) is 1600 train_network.py:1514
INFO skipping epoch 6 because initial_step (multiplied) is 1500 train_network.py:1514
INFO skipping epoch 7 because initial_step (multiplied) is 1400 train_network.py:1514
INFO skipping epoch 8 because initial_step (multiplied) is 1300 train_network.py:1514
INFO skipping epoch 9 because initial_step (multiplied) is 1200 train_network.py:1514
INFO skipping epoch 10 because initial_step (multiplied) is 1100 train_network.py:1514
INFO skipping epoch 11 because initial_step (multiplied) is 1000 train_network.py:1514
INFO skipping epoch 12 because initial_step (multiplied) is 900 train_network.py:1514
INFO skipping epoch 13 because initial_step (multiplied) is 800 train_network.py:1514
INFO skipping epoch 14 because initial_step (multiplied) is 700 train_network.py:1514
INFO skipping epoch 15 because initial_step (multiplied) is 600 train_network.py:1514
INFO skipping epoch 16 because initial_step (multiplied) is 500 train_network.py:1514
INFO skipping epoch 17 because initial_step (multiplied) is 400 train_network.py:1514
INFO skipping epoch 18 because initial_step (multiplied) is 300 train_network.py:1514
INFO skipping epoch 19 because initial_step (multiplied) is 200 train_network.py:1514
INFO skipping epoch 20 because initial_step (multiplied) is 100 train_network.py:1514
INFO skipping epoch 21 because initial_step (multiplied) is 0 train_network.py:1514
INFO skipping epoch 22 because initial_step (multiplied) is -100 train_network.py:1514
INFO skipping epoch 23 because initial_step (multiplied) is -200 train_network.py:1514
INFO skipping epoch 24 because initial_step (multiplied) is -300 train_network.py:1514
INFO skipping epoch 25 because initial_step (multiplied) is -400 train_network.py:1514
INFO skipping epoch 26 because initial_step (multiplied) is -500 train_network.py:1514
INFO skipping epoch 27 because initial_step (multiplied) is -600 train_network.py:1514
INFO skipping epoch 28 because initial_step (multiplied) is -700 train_network.py:1514
INFO skipping epoch 29 because initial_step (multiplied) is -800 train_network.py:1514
INFO skipping epoch 30 because initial_step (multiplied) is -900 train_network.py:1514
INFO skipping epoch 31 because initial_step (multiplied) is -1000 train_network.py:1514
INFO skipping epoch 32 because initial_step (multiplied) is -1100 train_network.py:1514
INFO skipping epoch 33 because initial_step (multiplied) is -1200 train_network.py:1514
INFO skipping epoch 34 because initial_step (multiplied) is -1300 train_network.py:1514
INFO skipping epoch 35 because initial_step (multiplied) is -1400 train_network.py:1514
INFO skipping epoch 36 because initial_step (multiplied) is -1500 train_network.py:1514
INFO skipping epoch 37 because initial_step (multiplied) is -1600 train_network.py:1514
INFO skipping epoch 38 because initial_step (multiplied) is -1700 train_network.py:1514
INFO skipping epoch 39 because initial_step (multiplied) is -1800 train_network.py:1514
INFO skipping epoch 40 because initial_step (multiplied) is -1900 train_network.py:1514
INFO skipping epoch 41 because initial_step (multiplied) is -2000 train_network.py:1514
INFO skipping epoch 42 because initial_step (multiplied) is -2100 train_network.py:1514
INFO skipping epoch 43 because initial_step (multiplied) is -2200 train_network.py:1514
INFO skipping epoch 44 because initial_step (multiplied) is -2300 train_network.py:1514
INFO skipping epoch 45 because initial_step (multiplied) is -2400 train_network.py:1514
INFO skipping epoch 46 because initial_step (multiplied) is -2500 train_network.py:1514
INFO skipping epoch 47 because initial_step (multiplied) is -2600 train_network.py:1514
INFO skipping epoch 48 because initial_step (multiplied) is -2700 train_network.py:1514
INFO skipping epoch 49 because initial_step (multiplied) is -2800 train_network.py:1514
INFO skipping epoch 50 because initial_step (multiplied) is -2900 train_network.py:1514
INFO skipping epoch 51 because initial_step (multiplied) is -3000 train_network.py:1514
INFO skipping epoch 52 because initial_step (multiplied) is -3100 train_network.py:1514
INFO skipping epoch 53 because initial_step (multiplied) is -3200 train_network.py:1514
INFO skipping epoch 54 because initial_step (multiplied) is -3300 train_network.py:1514
INFO skipping epoch 55 because initial_step (multiplied) is -3400 train_network.py:1514
INFO skipping epoch 56 because initial_step (multiplied) is -3500 train_network.py:1514
INFO skipping epoch 57 because initial_step (multiplied) is -3600 train_network.py:1514
INFO skipping epoch 58 because initial_step (multiplied) is -3700 train_network.py:1514
INFO skipping epoch 59 because initial_step (multiplied) is -3800 train_network.py:1514
INFO skipping epoch 60 because initial_step (multiplied) is -3900 train_network.py:1514
INFO skipping epoch 61 because initial_step (multiplied) is -4000 train_network.py:1514
INFO skipping epoch 62 because initial_step (multiplied) is -4100 train_network.py:1514
INFO skipping epoch 63 because initial_step (multiplied) is -4200 train_network.py:1514
INFO skipping epoch 64 because initial_step (multiplied) is -4300 train_network.py:1514
INFO skipping epoch 65 because initial_step (multiplied) is -4400 train_network.py:1514
INFO skipping epoch 66 because initial_step (multiplied) is -4500 train_network.py:1514
INFO skipping epoch 67 because initial_step (multiplied) is -4600 train_network.py:1514
INFO skipping epoch 68 because initial_step (multiplied) is -4700 train_network.py:1514
INFO skipping epoch 69 because initial_step (multiplied) is -4800 train_network.py:1514
INFO skipping epoch 70 because initial_step (multiplied) is -4900 train_network.py:1514
INFO skipping epoch 71 because initial_step (multiplied) is -5000 train_network.py:1514
INFO skipping epoch 72 because initial_step (multiplied) is -5100 train_network.py:1514
INFO skipping epoch 73 because initial_step (multiplied) is -5200 train_network.py:1514
INFO skipping epoch 74 because initial_step (multiplied) is -5300 train_network.py:1514
INFO skipping epoch 75 because initial_step (multiplied) is -5400 train_network.py:1514
INFO skipping epoch 76 because initial_step (multiplied) is -5500 train_network.py:1514
INFO skipping epoch 77 because initial_step (multiplied) is -5600 train_network.py:1514
INFO skipping epoch 78 because initial_step (multiplied) is -5700 train_network.py:1514
INFO skipping epoch 79 because initial_step (multiplied) is -5800 train_network.py:1514
INFO skipping epoch 80 because initial_step (multiplied) is -5900 train_network.py:1514
INFO unet dtype: torch.bfloat16, device: cuda:0 train_network.py:1519
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in train_network.py around lines 1461 and 1514, where resume steps are logged and epochs are skipped. Reproduce the reported case with --resume, --skip_until_initial_step, current_step=500, 100 batches per epoch, and gradient accumulation of 4. Done means 20 epochs are skipped rather than 80, the step count does not become negative, and training enters the loop.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 70/100