Broken training that appeared suddenly

I get this issue after training (I'm using kohya, but manually calling the training script:
```python
Traceback (most recent call last):
File "C:\Users\waxel\kohya\kohya_ss\sd-scripts\sdxl_train_network.py", line 184, in
trainer.train(args)
File "C:\Users\waxel\kohya\kohya_ss\sd-scripts\train_network.py", line 243, in train
vae.set_use_memory_efficient_attention_xformers(args.xformers)
File "C:\Users\waxel\kohya\kohya_ss\venv\lib\site-packages\diffusers\models\modeling_utils.py", line 262, in set_use_memory_efficient_attention_xformers
fn_recursive_set_mem_eff(module)
File "C:\Users\waxel\kohya\kohya_ss\venv\lib\site-packages\diffusers\models\modeling_utils.py", line 258, in fn_recursive_set_mem_eff
fn_recursive_set_mem_eff(child)
File "C:\Users\waxel\kohya\kohya_ss\venv\lib\site-packages\diffusers\models\modeling_utils.py", line 258, in fn_recursive_set_mem_eff
fn_recursive_set_mem_eff(child)
File "C:\Users\waxel\kohya\kohya_ss\venv\lib\site-packages\diffusers\models\modeling_utils.py", line 258, in fn_recursive_set_mem_eff
fn_recursive_set_mem_eff(child)
File "C:\Users\waxel\kohya\kohya_ss\venv\lib\site-packages\diffusers\models\modeling_utils.py", line 255, in fn_recursive_set_mem_eff
module.set_use_memory_efficient_attention_xformers(valid, attention_op)
File "C:\Users\waxel\kohya\kohya_ss\venv\lib\site-packages\diffusers\models\attention_processor.py", line 273, in set_use_memory_efficient_attention_xformers
raise e
File "C:\Users\waxel\kohya\kohya_ss\venv\lib\site-packages\diffusers\models\attention_processor.py", line 268, in set_use_memory_efficient_attention_xformers
torch.randn((1, 2, 40), device="cuda"),
RuntimeError: CUDA error: invalid argument
```

I'm training sdxl model as usual, there no problem, and suddenly it appeared (thow most of the time 4/5 it crashes but I just rerun and its fine, but here it crashes indefinetly, I tried like 30 times.)
In case I have not activated Memory efficient attention.

Start with sd-scripts/sdxl_train_network.py and train_network.py at the call to vae.set_use_memory_efficient_attention_xformers(args.xformers), then inspect the reported diffusers attention_processor.py failure. Reproduce the SDXL training run with the same xformers setting and CUDA environment; done means identifying why the invalid CUDA argument occurs and documenting or validating a fix.

Written by the indexing model from the issue text.