error on dreambooth TI with a geforce 4090

i'm new to this, and trying to set everything up, in the process i got to the point where i want to start a "Dreambooth TI", i setup everything and when i click on "Train Ti" in the end, i get the follwoing error:

i'm on a nvidia geforce 4090 here.
one idea that we got here is that this seems to run cuda 11.6 and not 11.8

here comes the error:

1 train images with repeating.
use template for training captions. is object: {args.use_object_template}
loading image sizes.
100%|███████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 166.53it/s]
prepare dataset
Replace CrossAttention.forward to use xformers
prepare optimizer, data loader etc.
49416 tensor(49408)
running training / 学習開始
num train images * repeats / 学習画像の数×繰り返し回数: 1
num reg images / 正則化画像の数: 0
num batches per epoch / 1epochのバッチ数: 1
num epochs / epoch数: 100
batch size per device / バッチサイズ: 1
total train batch size (with parallel & distributed & accumulation) / 総バッチサイズ（並列学習、勾配合計含む）: 1
gradient ccumulation steps / 勾配を合計するステップ数 = 1
total optimization steps / 学習ステップ数: 100
steps: 0%| | 0/100 [00:00
train(args)
File "C:\Programmet\AI\kohya_ss\train_textual_inversion.py", line 345, in train
accelerator.backward(loss)
File "C:\programmet\ai\kohya_ss\venv\lib\site-packages\accelerate\accelerator.py", line 1316, in backward
loss.backward(**kwargs)
File "C:\programmet\ai\kohya_ss\venv\lib\site-packages\torch\_tensor.py", line 396, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "C:\programmet\ai\kohya_ss\venv\lib\site-packages\torch\autograd\__init__.py", line 173, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
File "C:\programmet\ai\kohya_ss\venv\lib\site-packages\torch\autograd\function.py", line 253, in apply
return user_fn(self, *args)
File "C:\programmet\ai\kohya_ss\venv\lib\site-packages\xformers\ops.py", line 369, in backward
) = torch.ops.xformers.efficient_attention_backward_cutlass(
File "C:\programmet\ai\kohya_ss\venv\lib\site-packages\torch\_ops.py", line 143, in __call__
return self._op(*args, **kwargs or {})
RuntimeError: CUDA error: invalid argument
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
steps: 0%| | 0/100 [00:44
File "C:\programmet\ai\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 45, in main
args.func(args)
File "C:\programmet\ai\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1104, in launch_command
simple_launcher(args)
File "C:\programmet\ai\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 567, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['C:\\programmet\\ai\\kohya_ss\\venv\\Scripts\\python.exe', 'train_textual_inversion.py', '--pretrained_model_name_or_path=D:\\stable-diffusion-webui\\models\\Stable-diffusion\\v1-5-pruned-emaonly.safetensors', '--train_data_dir=C:\\Programmet\\AI\\kohya_ss\\output\\training\\img', '--resolution=512,512', '--output_dir=C:\\Programmet\\AI\\kohya_ss\\output\\training\\model', '--logging_dir=C:\\Programmet\\AI\\kohya_ss\\output\\training\\log', '--save_model_as=safetensors', '--output_name=test', '--learning_rate=1e-3', '--lr_scheduler=cosine', '--lr_warmup_steps=10', '--train_batch_size=1', '--max_train_steps=100', '--save_every_n_epochs=10', '--mixed_precision=no', '--save_precision=fp16', '--seed=1234', '--xformers', '--token_string=malicor', '--init_word=*', '--num_vectors_per_token=8', '--use_object_template']' returned non-zero exit status 1.

Start with train_textual_inversion.py, especially train(args) around the accelerator.backward(loss) call, and inspect the xformers efficient_attention_backward_cutlass failure shown in the traceback. Reproduce the Dreambooth TI command on the reported GeForce 4090 setup; the issue is resolved when training no longer exits with CUDA error: invalid argument.

Written by the indexing model from the issue text.