kohya-ss / kohya-ss/sd-scripts
Resuming training from a state that was resumed training from earlier state behaves weird
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 1.2k
- Avg merge
- 11m
- Merged PRs (30d)
- 2
Description
When resuming training from a training state that was also created from resuming training from a training state behaves weird.
Example: 5 epochs. Save state every epoch.
* Blue line: Normal training from start to finish.
* Red line: Resume from state of epoch 2. `resume = "E:/training/output/test_1-000002-state"`
* yellowline: Resumed from first saved state (epoch 3) of the previously resumed training `resume = "E:/training/output/test_2-000003-state"`
The first resumed training (redline) trains for 3 epochs and finishes at total of 2+3=5 epochs as expected.
The resumed-resumed training (yellowline) trains for 4 epochs resulting in of 2+1+4=7 epochs of training.
This may be as simple as the `"current_step"` being saved with wrong number in train_state.json. But I am not good enough to know if that is the problem.
Used training settings: [training_tomls.zip](https://github.com/user-attachments/files/21706249/training_tomls.zip)
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the nested-resume case using the training settings in training_tomls.zip: resume from epoch 2, then resume again from the saved epoch-3 state. Inspect train_state.json, especially current_step, and compare the resulting epoch counts with the reported 5, 5, and 7 epochs. Done means nested resume training stops at the expected total rather than adding extra epochs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100