kohya-ss / kohya-ss/sd-scripts
Failed to resume from state
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 1.2k
- Avg merge
- 11m
- Merged PRs (30d)
- 2
Description
```
load train state from /home/ubuntu/MyFiles/xiaodi/training/output/flux/sd-scripts/flux_lora_200k/flux_lora_200k-step00002250-state/train_state.json: {'current_epoch': 1, 'current_step': 2250}
[rank0]: Traceback (most recent call last):
[rank0]: File "/home/ubuntu/sd-scripts/flux_train_network.py", line 411, in
[rank0]: trainer.train(args)
[rank0]: File "/home/ubuntu/sd-scripts/train_network.py", line 663, in train
[rank0]: train_util.resume_from_local_or_hf_if_specified(accelerator, args)
[rank0]: File "/home/ubuntu/sd-scripts/library/train_util.py", line 4260, in resume_from_local_or_hf_if_specified
[rank0]: accelerator.load_state(args.resume)
[rank0]: File "/home/ubuntu/miniconda3/envs/sd-scripts/lib/python3.11/site-packages/accelerate/accelerator.py", line 3145, in load_state
[rank0]: override_attributes = load_accelerator_state(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/ubuntu/miniconda3/envs/sd-scripts/lib/python3.11/site-packages/accelerate/checkpointing.py", line 208, in load_accelerator_state
[rank0]: load_model(model, input_model_file, device=str(map_location), **load_model_func_kwargs)
[rank0]: TypeError: load_model() got an unexpected keyword argument 'device'
```
It caused by the wrong version `safetensors==0.4.2` in `requirements.txt`, safetensors add this argument in this commit https://github.com/huggingface/safetensors/commit/ff643a874414bf976ebe6857c59320f1e8f4e4b4, upgrade to `safetensors==0.4.4` solved this problem.
another error occupied is
```
[rank2]: Traceback (most recent call last):
[rank2]: File "/home/ubuntu/sd-scripts/flux_train_network.py", line 411, in
[rank2]: trainer.train(args)
[rank2]: File "/home/ubuntu/sd-scripts/train_network.py", line 663, in train
[rank2]: train_util.resume_from_local_or_hf_if_specified(accelerator, args)
[rank2]: File "/home/ubuntu/sd-scripts/library/train_util.py", line 4260, in resume_from_local_or_hf_if_specified
[rank2]: accelerator.load_state(args.resume)
[rank2]: File "/home/ubuntu/miniconda3/envs/sd-scripts/lib/python3.11/site-packages/accelerate/accelerator.py", line 3156, in load_state
[rank2]: self.step = override_attributes["step"]
```
downgrade accelerate to `accelerate==0.31.0` solved this problem
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with requirements.txt and inspect the pinned safetensors and accelerate versions against the resume errors reported in flux_train_network.py and train_network.py. Update the dependency constraints based on the versions identified in the issue, then verify that resuming from a saved train_state.json no longer raises either traceback.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100