kohya-ss / kohya-ss/sd-scripts

Resuming training from a state that was resumed training from earlier state behaves weird

Open
#2,171 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

When resuming training from a training state that was also created from resuming training from a training state behaves weird.

Example: 5 epochs. Save state every epoch.
* Blue line: Normal training from start to finish.
* Red line: Resume from state of epoch 2. `resume = "E:/training/output/test_1-000002-state"`
* yellowline: Resumed from first saved state (epoch 3) of the previously resumed training `resume = "E:/training/output/test_2-000003-state"`
Image

The first resumed training (redline) trains for 3 epochs and finishes at total of 2+3=5 epochs as expected.
The resumed-resumed training (yellowline) trains for 4 epochs resulting in of 2+1+4=7 epochs of training.

This may be as simple as the `"current_step"` being saved with wrong number in train_state.json. But I am not good enough to know if that is the problem.

Used training settings: [training_tomls.zip](https://github.com/user-attachments/files/21706249/training_tomls.zip)

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the nested-resume case using the training settings in training_tomls.zip: resume from epoch 2, then resume again from the saved epoch-3 state. Inspect train_state.json, especially current_step, and compare the resulting epoch counts with the reported 5, 5, and 7 epochs. Done means nested resume training stops at the expected total rather than adding extra epochs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.