kohya-ss / kohya-ss/sd-scripts

[bug] resume_from is ignored in sdxl_train_textual_inversion.py

Open
#2,132 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

I'm encountering a critical issue with the `resume_from` parameter when training a *textual inversion* embedding on **Stable Diffusion XL** using `sdxl_train_textual_inversion.py`.

Despite specifying a valid `.safetensors` file in the `resume_from` field (with proper token name and path), the training **starts from scratch**: it initializes the embedding as if no prior progress existed.

**Context:**
- I’m using the bmaltais-style config (`.toml`), launched with:
`accelerate launch ... sdxl_train_textual_inversion.py --config_file=/path/to/file.toml`
- The file specified in `resume_from` exists, is readable, and contains a learned token trained for ~2000 steps
- Training resumes at **step 0**, with no error, and the loss is identical to the initial run (~0.105)
- I also tried `pretrained_embedding_path` — same behavior
- I cannot use `resume_from` to continue a long training (10k+ steps), making any interruption fatal

**Expected behavior:**
Training resumes correctly from the `.safetensors` file, restoring the learned vectors and optimizer state.

**Actual behavior:**
The script silently ignores the checkpoint and resets all training state (including token weights and optimizer).

**To reproduce:**
1. Train a textual inversion embedding with SDXL
2. Save a `.safetensors` at step N
3. Stop training
4. Restart with `resume_from = "path/to/embedding-N.safetensors"`
5. Observe that training restarts from step 0, reinitializes embedding, and behaves like a fresh run

**System:**
- GPU: AMD RX 7900 XTX (ROCm 6.3.2)
- Tested with: latest main branch of `sd-scripts`
- Config: bmaltais `kohya_ss` style config + `sdxl_train_textual_inversion.py`

Let me know if you need logs, test cases, or can guide me to patch this. Thanks!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in sdxl_train_textual_inversion.py and trace how resume_from and pretrained_embedding_path are read when the bmaltais-style TOML config is passed. Reproduce the documented stop-and-restart sequence with a .safetensors embedding, then verify that training does not return to step 0 or reinitialize the learned token and training state.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.