kohya-ss / kohya-ss/sd-scripts
Flux Dreambooth crashes after trying to save checkpoint
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 1.2k
- Avg merge
- 11m
- Merged PRs (30d)
- 2
Description
I am able to successful db training and samples look good, but after 600 steps it gives me this error
2024-08-19 07:57:06 INFO train_util.py:5123
INFO saving checkpoint: train_util.py:5124
/home/Ubuntu/Downloads/model
4/Rick-step00000600.safetensors
Traceback (most recent call last):
File "/home/Ubuntu/apps/kohya_ss/venv/bin/accelerate", line 8, in
sys.exit(main())
File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
args.func(args)
File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1106, in launch_command
simple_launcher(args)
File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 704, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/Ubuntu/apps/kohya_ss/venv/bin/python', '/home/Ubuntu/apps/kohya_ss/sd-scripts/flux_train.py', '--config_file', '/home/Ubuntu/Downloads/model 4/config_dreambooth-20240819-073001.toml', '--fp8_base', '--highvram', '--cpu_offload_checkpointing']' died with .
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the checkpoint-saving path around train_util.py:5123-5124 and the flux_train.py entry point, using the shown config_dreambooth TOML and the --cpu_offload_checkpointing options. Reproduce the run through step 600 and determine why the process receives SIGKILL during checkpoint saving; done means the checkpoint is written without that termination.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100