kohya-ss / kohya-ss/sd-scripts

Flux Dreambooth crashes after trying to save checkpoint

Open
#1,476 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

I am able to successful db training and samples look good, but after 600 steps it gives me this error

2024-08-19 07:57:06 INFO train_util.py:5123
INFO saving checkpoint: train_util.py:5124
/home/Ubuntu/Downloads/model
4/Rick-step00000600.safetensors
Traceback (most recent call last):
File "/home/Ubuntu/apps/kohya_ss/venv/bin/accelerate", line 8, in
sys.exit(main())
File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
args.func(args)
File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1106, in launch_command
simple_launcher(args)
File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 704, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/Ubuntu/apps/kohya_ss/venv/bin/python', '/home/Ubuntu/apps/kohya_ss/sd-scripts/flux_train.py', '--config_file', '/home/Ubuntu/Downloads/model 4/config_dreambooth-20240819-073001.toml', '--fp8_base', '--highvram', '--cpu_offload_checkpointing']' died with .

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the checkpoint-saving path around train_util.py:5123-5124 and the flux_train.py entry point, using the shown config_dreambooth TOML and the --cpu_offload_checkpointing options. Reproduce the run through step 600 and determine why the process receives SIGKILL during checkpoint saving; done means the checkpoint is written without that termination.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.