kohya-ss / kohya-ss/sd-scripts

Failed to resume from state

Open
#1,524 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

```
load train state from /home/ubuntu/MyFiles/xiaodi/training/output/flux/sd-scripts/flux_lora_200k/flux_lora_200k-step00002250-state/train_state.json: {'current_epoch': 1, 'current_step': 2250}
[rank0]: Traceback (most recent call last):
[rank0]: File "/home/ubuntu/sd-scripts/flux_train_network.py", line 411, in
[rank0]: trainer.train(args)
[rank0]: File "/home/ubuntu/sd-scripts/train_network.py", line 663, in train
[rank0]: train_util.resume_from_local_or_hf_if_specified(accelerator, args)
[rank0]: File "/home/ubuntu/sd-scripts/library/train_util.py", line 4260, in resume_from_local_or_hf_if_specified
[rank0]: accelerator.load_state(args.resume)
[rank0]: File "/home/ubuntu/miniconda3/envs/sd-scripts/lib/python3.11/site-packages/accelerate/accelerator.py", line 3145, in load_state
[rank0]: override_attributes = load_accelerator_state(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/ubuntu/miniconda3/envs/sd-scripts/lib/python3.11/site-packages/accelerate/checkpointing.py", line 208, in load_accelerator_state
[rank0]: load_model(model, input_model_file, device=str(map_location), **load_model_func_kwargs)
[rank0]: TypeError: load_model() got an unexpected keyword argument 'device'
```
It caused by the wrong version `safetensors==0.4.2` in `requirements.txt`, safetensors add this argument in this commit https://github.com/huggingface/safetensors/commit/ff643a874414bf976ebe6857c59320f1e8f4e4b4, upgrade to `safetensors==0.4.4` solved this problem.

another error occupied is
```
[rank2]: Traceback (most recent call last):
[rank2]: File "/home/ubuntu/sd-scripts/flux_train_network.py", line 411, in
[rank2]: trainer.train(args)
[rank2]: File "/home/ubuntu/sd-scripts/train_network.py", line 663, in train
[rank2]: train_util.resume_from_local_or_hf_if_specified(accelerator, args)
[rank2]: File "/home/ubuntu/sd-scripts/library/train_util.py", line 4260, in resume_from_local_or_hf_if_specified
[rank2]: accelerator.load_state(args.resume)
[rank2]: File "/home/ubuntu/miniconda3/envs/sd-scripts/lib/python3.11/site-packages/accelerate/accelerator.py", line 3156, in load_state
[rank2]: self.step = override_attributes["step"]
```
downgrade accelerate to `accelerate==0.31.0` solved this problem

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with requirements.txt and inspect the pinned safetensors and accelerate versions against the resume errors reported in flux_train_network.py and train_network.py. Update the dependency constraints based on the versions identified in the issue, then verify that resuming from a saved train_state.json no longer raises either traceback.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.