kohya-ss / kohya-ss/sd-scripts

How to resume interrupted training for Flux?

Open
#1,520 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

I'm using the following command for training:

``` bash
accelerate launch --mixed_precision bf16 --num_cpu_threads_per_process 1 flux_train_network.py --pretrained_model_name_or_path ~/work/media/flux_lora/models/flux1-dev.safetensors --clip_l ~/work/media/flux_lora/models/clip_l.safetensors --t5xxl ~/work/media/flux_lora/models/t5xxl_fp16.safetensors --ae ~/work/media/flux_lora/models/ae.safetensors --cache_latents_to_disk --save_model_as safetensors --sdpa --persistent_data_loader_workers --max_data_loader_n_workers 2 --seed 42 --gradient_checkpointing --mixed_precision bf16 --save_precision bf16 --network_module networks.lora_flux --network_dim 4 --optimizer_type adamw8bit --learning_rate 1e-4 --network_train_unet_only --cache_text_encoder_outputs --cache_text_encoder_outputs_to_disk --fp8_base --highvram --max_train_epochs 64 --save_every_n_epochs 4 --dataset_config dataset.toml --output_dir ~/work/media/flux_lora/training-output --output_name flux-lora-output --timestep_sampling sigmoid --model_prediction_type raw --guidance_scale 1.0 --loss_type l2
```

If training is interrupted, running this command again starts from scratch. How can I modify the command or process to resume training from the last checkpoint instead of starting over?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with flux_train_network.py and the supplied training command, especially the save_every_n_epochs and output_dir settings. Check how interrupted Flux runs and saved checkpoints are handled, then document the command or process needed to continue from the latest checkpoint. Done means a reader can resume training without restarting from scratch.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation, machine-learning
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.