kohya-ss / kohya-ss/sd-scripts

Issues when restarting training session

Open
#2,248 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

Preface: This is actually happening on a GUI using sd-scripts as backend. After talking to the maintainer, he told me it's an issue related to sd-scripts, I came to ask here if the issue is actually related to sd-script and if you faced similar issues, if it's possible to fix this issue.

**Describe the bug**
I've been doing longer training sessions testing multiple styles lora. I noticed that if I reasume a state more than once I don't get the correct number of steps and epoch.
I'm not sure if this can create issues for scheduler or warmup.

Example:
I want to train a total of 30 epoch.
I train 1000 steps 10 epoch.
I stop the training, epoch 10 state saved.
I reassure the training from epoch 10, everything seems correct.
I train another 1500 steps and 15 epoch and keep the last save.
I resume the state, but somehow I'm restarting from 1500 steps 15 epoch, instead of epoch 25.

**Expected behavior**
Steps and epoch should stockpile during all sequential training sessions. Doesn't seem to do so.

**How to recreate it**
Just get two different successive save states, they won't sum. Just a few steps to see the issue.

Thanks

Contributor guide

No contributing guide indexed for this repository

Research direction

The report names no source file, test, or entry point. First reproduce the two successive save-and-resume sessions described in the issue, then trace how resumed steps and epochs are calculated. Done means successive checkpoints report cumulative steps and epochs, with scheduler or warmup behavior checked.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.