kohya-ss / kohya-ss/sd-scripts
How to resume interrupted training for Flux?
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 1.2k
- Avg merge
- 11m
- Merged PRs (30d)
- 2
Description
I'm using the following command for training:
``` bash
accelerate launch --mixed_precision bf16 --num_cpu_threads_per_process 1 flux_train_network.py --pretrained_model_name_or_path ~/work/media/flux_lora/models/flux1-dev.safetensors --clip_l ~/work/media/flux_lora/models/clip_l.safetensors --t5xxl ~/work/media/flux_lora/models/t5xxl_fp16.safetensors --ae ~/work/media/flux_lora/models/ae.safetensors --cache_latents_to_disk --save_model_as safetensors --sdpa --persistent_data_loader_workers --max_data_loader_n_workers 2 --seed 42 --gradient_checkpointing --mixed_precision bf16 --save_precision bf16 --network_module networks.lora_flux --network_dim 4 --optimizer_type adamw8bit --learning_rate 1e-4 --network_train_unet_only --cache_text_encoder_outputs --cache_text_encoder_outputs_to_disk --fp8_base --highvram --max_train_epochs 64 --save_every_n_epochs 4 --dataset_config dataset.toml --output_dir ~/work/media/flux_lora/training-output --output_name flux-lora-output --timestep_sampling sigmoid --model_prediction_type raw --guidance_scale 1.0 --loss_type l2
```
If training is interrupted, running this command again starts from scratch. How can I modify the command or process to resume training from the last checkpoint instead of starting over?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with flux_train_network.py and the supplied training command, especially the save_every_n_epochs and output_dir settings. Check how interrupted Flux runs and saved checkpoints are handled, then document the command or process needed to continue from the latest checkpoint. Done means a reader can resume training without restarting from scratch.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation, machine-learning
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100