kohya-ss / kohya-ss/sd-scripts
Validation isn't working in Dreambooth
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 1.2k
- Avg merge
- 11m
- Merged PRs (30d)
- 2
Description
I'm trying to train Flux in the Dreambooth tab and can't get validation to work. I'm using Runpod. Normally, I train LoRAs locally and validation works fine. It seems to detect my validation set just fine, but once it gets to 'running training' there is no 'num validation images '.
```
2025-06-12 20:27:51 INFO Loading settings from /workspace/Models/diffusion_models/rp2/config_dreambooth-20250612.toml... train_util.py:4651
2025-06-12 20:27:51 INFO Load dataset config from /workspace/dataset.toml flux_train.py:104
INFO prepare images. train_util.py:2072
INFO found directory /workspace/img1 contains 50 image files train_util.py:2019
read caption: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:00<00:00, 501.00it/s]
INFO 50 train images with repeats. train_util.py:2116
INFO 0 reg images with repeats. train_util.py:2120
WARNING no regularization images / 正則化画像が見つかりませんでした train_util.py:2125
INFO prepare images. train_util.py:2072
INFO found directory /workspace/img2 man contains 26 image files train_util.py:2019
read caption: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 26/26 [00:00<00:00, 506.70it/s]
INFO 26 train images with repeats. train_util.py:2116
INFO 0 reg images with repeats. train_util.py:2120
WARNING no regularization images / 正則化画像が見つかりませんでした train_util.py:2125
INFO prepare images. train_util.py:2072
INFO found directory /workspace/val contains 0 image files train_util.py:2019
read caption: 0it [00:00, ?it/s]
WARNING ignore subset with image_dir='/workspace/val': no images found / train_util.py:2092
画像が見つからないためサブセットを無視します
INFO 0 train images with repeats. train_util.py:2116
INFO 0 reg images with repeats. train_util.py:2120
WARNING no regularization images / 正則化画像が見つかりませんでした train_util.py:2125
INFO prepare images. train_util.py:2072
INFO found directory /workspace/val contains 6 image files train_util.py:2019
read caption: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 505.21it/s]
INFO 6 validation images with repeats. train_util.py:2116
INFO 0 reg images with repeats. train_util.py:2120
WARNING no regularization images / 正則化画像が見つかりませんでした train_util.py:2125
INFO [Dataset 0] config_util.py:580
batch_size: 1
resolution: (512, 512)
resize_interpolation: None
enable_bucket: False
[Subset 0 of Dataset 0]
image_dir: "/workspace/img1"
image_count: 50
num_repeats: 1
shuffle_caption: False
keep_tokens: 0
caption_dropout_rate: 0
caption_dropout_every_n_epochs: 0
caption_tag_dropout_rate: 0.0
caption_prefix: None
caption_suffix: None
color_aug: False
flip_aug: False
face_crop_aug_range: None
random_crop: True
token_warmup_min: 1,
token_warmup_step: 0,
alpha_mask: False
resize_interpolation: None
custom_attributes: {}
is_reg: False
class_tokens: None
caption_extension: .txt
[Dataset 1]
batch_size: 1
resolution: (1024, 1024)
resize_interpolation: None
enable_bucket: False
[Subset 0 of Dataset 1]
image_dir: "/workspace/img2 man"
image_count: 26
num_repeats: 1
shuffle_caption: False
keep_tokens: 0
caption_dropout_rate: 0
caption_dropout_every_n_epochs: 0
caption_tag_dropout_rate: 0.0
caption_prefix: None
caption_suffix: None
color_aug: False
flip_aug: False
face_crop_aug_range: None
random_crop: True
token_warmup_min: 1,
token_warmup_step: 0,
alpha_mask: False
resize_interpolation: None
custom_attributes: {}
is_reg: False
class_tokens: None
caption_extension: .txt
[Dataset 2]
batch_size: 1
resolution: (512, 512)
resize_interpolation: None
enable_bucket: False
INFO [Validation Dataset 0] config_util.py:580
batch_size: 1
resolution: (512, 512)
resize_interpolation: None
enable_bucket: False
[Subset 0 of Validation Dataset 0]
image_dir: "/workspace/val"
image_count: 6
num_repeats: 1
shuffle_caption: False
keep_tokens: 0
caption_dropout_rate: 0
caption_dropout_every_n_epochs: 0
caption_tag_dropout_rate: 0.0
caption_prefix: None
caption_suffix: None
color_aug: False
flip_aug: False
face_crop_aug_range: None
random_crop: False
token_warmup_min: 1,
token_warmup_step: 0,
alpha_mask: False
resize_interpolation: None
custom_attributes: {}
is_reg: False
class_tokens: None
caption_extension: .txt
INFO [Prepare dataset 0] config_util.py:592
INFO loading image sizes. train_util.py:987
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:00<00:00, 484.04it/s]
INFO prepare dataset train_util.py:1012
INFO [Prepare dataset 1] config_util.py:592
INFO loading image sizes. train_util.py:987
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 26/26 [00:00<00:00, 524.12it/s]
INFO prepare dataset train_util.py:1012
INFO [Prepare dataset 2] config_util.py:592
INFO loading image sizes. train_util.py:987
0it [00:00, ?it/s]
INFO prepare dataset train_util.py:1012
INFO [Prepare validation dataset 0] config_util.py:597
INFO loading image sizes. train_util.py:987
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 556.75it/s]
INFO prepare dataset train_util.py:1012
INFO Checking the state dict: Diffusers or BFL, dev or schnell flux_utils.py:43
INFO prepare accelerator flux_train.py:186
accelerator device: cuda
/workspace/kohya_ss/venv/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884
warnings.warn(
You are using the default legacy behaviour of the . This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
2025-06-12 20:27:52 INFO Building CLIP-L flux_utils.py:179
INFO Loading state dict from /workspace/clip_l.safetensors flux_utils.py:275
INFO Loaded CLIP-L: flux_utils.py:278
INFO Loading state dict from /workspace/t5xxl_fp16.safetensors flux_utils.py:330
2025-06-12 20:27:57 INFO Loaded T5xxl: flux_utils.py:333
2025-06-12 20:27:58 INFO [Dataset 0] train_util.py:2635
INFO caching Text Encoder outputs with caching strategy. train_util.py:1298
INFO checking cache validity... train_util.py:1309
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:00<00:00, 5428.68it/s]
INFO caching Text Encoder outputs... train_util.py:1340
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:04<00:00, 10.59it/s]
2025-06-12 20:28:03 INFO [Dataset 1] train_util.py:2635
INFO caching Text Encoder outputs with caching strategy. train_util.py:1298
INFO checking cache validity... train_util.py:1309
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 26/26 [00:00<00:00, 413.86it/s]
INFO no Text Encoder outputs to cache train_util.py:1336
INFO [Dataset 2] train_util.py:2635
INFO caching Text Encoder outputs with caching strategy. train_util.py:1298
INFO checking cache validity... train_util.py:1309
0it [00:00, ?it/s]
INFO no Text Encoder outputs to cache train_util.py:1336
INFO cache Text Encoder outputs for sample prompt: /workspace/Models/diffusion_models/rp2/sample/prompt.txt flux_train.py:249
INFO cache Text Encoder outputs for prompt: prmpt1 flux_train.py:259
prmpt1
INFO cache Text Encoder outputs for prompt: flux_train.py:259
INFO cache Text Encoder outputs for prompt: prmpt2 flux_train.py:259
prmpt2
INFO cache Text Encoder outputs for prompt: prmpt3 flux_train.py:259
prmpt3
INFO cache Text Encoder outputs for prompt: prmpt4 flux_train.py:259
2025-06-12 20:28:04 INFO Checking the state dict: Diffusers or BFL, dev or schnell flux_utils.py:43
INFO Building Flux model dev from BFL checkpoint flux_utils.py:101
INFO Loading state dict from /workspace/flux1-dev.safetensors flux_utils.py:118
INFO Loaded Flux: flux_utils.py:137
FLUX: Gradient checkpointing enabled. CPU offload: False
INFO Building AutoEncoder flux_utils.py:144
INFO Loading state dict from /workspace/ae.safetensors flux_utils.py:149
INFO Loaded AE: flux_utils.py:152
number of trainable parameters: 11901408320
prepare optimizer, data loader etc.
INFO use Adafactor optimizer | {'scale_parameter': False, 'relative_step': False, 'warmup_init': False, 'weight_decay': 0.01} train_util.py:4963
WARNING constant_with_warmup will be good / スケジューラはconstant_with_warmupが良いかもしれません train_util.py:4995
enable full bf16 training.
running training / 学習開始
num examples / サンプル数: 76
num batches per epoch / 1epochのバッチ数: 76
num epochs / epoch数: 132
batch size per device / バッチサイズ: 1, 1, 1
gradient accumulation steps / 勾配を合計するステップ数 = 1
total optimization steps / 学習ステップ数: 10000
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the Dreambooth validation run from the attached log and inspect validation dataset handling in train_util.py and config_util.py. Trace the transition into training in flux_train.py, using the logged validation dataset and image counts as checkpoints. Done means a configured validation set remains available during training and its image count is reported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100