Training of LoRA suddenly not working...

I got a problem suddenly when I used the same setting to train LoRA which helped me successfully train several models in previous week.
Here are the error messages: ( I have tried to reinstall the Python and update the pip with pip install -r requirements.txt, but the errors are still here...). So grateful to have advices to resolve the problem. Thanks!

Using decoupled weight decay
enable fp8 training.
running training / 学習開始
num train images * repeats / 学習画像の数×繰り返し回数: 320
num reg images / 正則化画像の数: 117
num batches per epoch / 1epochのバッチ数: 320
num epochs / epoch数: 13
batch size per device / バッチサイズ: 2
gradient accumulation steps / 勾配を合計するステップ数 = 1
total optimization steps / 学習ステップ数: 4000
steps: 0%| | 0/4000 [00:00
trainer.train(args)
File "B:\Kohya\kohya_ss\sd-scripts\train_network.py", line 804, in train
for step, batch in enumerate(train_dataloader):
File "B:\Kohya\kohya_ss\venv\lib\site-packages\accelerate\data_loader.py", line 458, in __iter__
next_batch = next(dataloader_iter)
File "B:\Kohya\kohya_ss\venv\lib\site-packages\torch\utils\data\dataloader.py", line 630, in __next__
data = self._next_data()
File "B:\Kohya\kohya_ss\venv\lib\site-packages\torch\utils\data\dataloader.py", line 1345, in _next_data
return self._process_data(data)
File "B:\Kohya\kohya_ss\venv\lib\site-packages\torch\utils\data\dataloader.py", line 1371, in _process_data
data.reraise()
File "B:\Kohya\kohya_ss\venv\lib\site-packages\torch\_utils.py", line 694, in reraise
raise exception
RuntimeError: Caught RuntimeError in DataLoader worker process 5.
Original Traceback (most recent call last):
File "B:\Kohya\kohya_ss\venv\lib\site-packages\torch\utils\data\_utils\worker.py", line 308, in _worker_loop
data = fetcher.fetch(index)
File "B:\Kohya\kohya_ss\venv\lib\site-packages\torch\utils\data\_utils\fetch.py", line 51, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "B:\Kohya\kohya_ss\venv\lib\site-packages\torch\utils\data\_utils\fetch.py", line 51, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "B:\Kohya\kohya_ss\venv\lib\site-packages\torch\utils\data\dataset.py", line 302, in __getitem__
return self.datasets[dataset_idx][sample_idx]
File "B:\Kohya\kohya_ss\sd-scripts\library\train_util.py", line 1341, in __getitem__
example["latents"] = torch.stack(latents_list) if latents_list[0] is not None else None
RuntimeError: stack expects each tensor to be equal size, but got [4, 128, 128] at entry 0 and [4, 124, 124] at entry 1

steps: 0%|▏ | 12/4000 [01:00<5:32:28, 5.00s/it, avr_loss=0.15]
Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "B:\Kohya\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 7, in
File "B:\Kohya\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 47, in main
args.func(args)
File "B:\Kohya\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1017, in launch_command
simple_launcher(args)
File "B:\Kohya\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 637, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['B:\\Kohya\\kohya_ss\\venv\\Scripts\\python.exe', 'B:/Kohya/kohya_ss/sd-scripts/sdxl_train_network.py', '--config_file', 'B:/Kohya/kohya_ss/outputs/Cary2_leosamsHelloworld_Park Min-young_4000\\model/config_lora-20240605-182340.toml', '--log_prefix=xl-locon', '--enable_wildcard']' returned non-zero exit status 1.
18:25:33-129584 INFO Training has ended.

Start with the reported traceback in sdxl_train_network.py, train_network.py, and library/train_util.py, then inspect the referenced training configuration file. Reproduce the failure and determine why the DataLoader receives latent tensors with different spatial sizes; done means training proceeds without the stack-size error and the cause or required configuration change is documented.

Written by the indexing model from the issue text.