dylora broken, ValueError: Attempting to unscale FP16 gradients

Start of training, SDXL example, to show that accelerate and the script loads model, UNet, VAE and everything. Ignore the warning about scale_weight_norms since it's turned off automatically.

```
2024-04-30 11:37:50 INFO loading model for process 0/1 sdxl_train_util.py:30
INFO load StableDiffusion sdxl_train_util.py:70
checkpoint:
/media/santi-linux/Linux_Part
_2/models/ponyDiffusionV6XL_v
6StartWithThisOne.safetensors
2024-04-30 11:37:51 INFO building U-Net sdxl_model_util.py:192
INFO loading U-Net from sdxl_model_util.py:196
checkpoint
2024-04-30 11:37:57 INFO U-Net:
2024-04-30 11:37:58 INFO building text encoders sdxl_model_util.py:205
INFO loading text encoders from sdxl_model_util.py:258
checkpoint
INFO text encoder 1:
2024-04-30 11:38:00 INFO text encoder 2:
INFO building VAE sdxl_model_util.py:279
INFO loading VAE from checkpoint sdxl_model_util.py:284
2024-04-30 11:38:01 INFO VAE:
2024-04-30 11:38:04 INFO Enable memory efficient train_util.py:2657
attention for U-Net
import network module: networks.dylora
2024-04-30 11:38:05 INFO [Dataset 0] train_util.py:2079
INFO caching latents. train_util.py:974
INFO checking cache validity... train_util.py:984
100%|████████████████████████████████████████| 252/252 [00:00<00:00, 422.51it/s]
INFO caching latents... train_util.py:1021
0it [00:00, ?it/s]
2024-04-30 11:38:06 INFO create LoRA network. base dim (rank): dylora.py:286
16, alpha: 16, unit: 4
INFO create LoRA for Text Encoder 1 dylora.py:330
INFO create LoRA for Text Encoder 2 dylora.py:330
INFO create LoRA for Text Encoder: 264 dylora.py:339
modules.
2024-04-30 11:38:07 INFO create LoRA for U-Net: 722 modules. dylora.py:347
WARNING warning: scale_weight_norms is train_network.py:316
specified but the network does
not support it /
scale_weight_normsが指定されて
いますが、ネットワークが対応し
ていません
INFO enable LoRA for text encoder dylora.py:367
INFO enable LoRA for U-Net dylora.py:372
prepare optimizer, data loader etc.
2024-04-30 11:38:08 INFO use 8-bit AdamW optimizer | {} train_util.py:3889
override steps. steps for 5 epochs is / 指定エポックまでのステップ数: 2230
running training / 学習開始
num train images * repeats / 学習画像の数×繰り返し回数: 446
num reg images / 正則化画像の数: 0
num batches per epoch / 1epochのバッチ数: 446
num epochs / epoch数: 5
batch size per device / バッチサイズ: 1
gradient accumulation steps / 勾配を合計するステップ数 = 1
total optimization steps / 学習ステップ数: 2230
steps: 0%| | 0/2230 [00:00
trainer.train(args)
File "/home/santi-linux/trainer_kohya_ss/sd-scripts/train_network.py", line 907, in train
accelerator.clip_grad_norm_(params_to_clip, args.max_grad_norm)
File "/home/santi-linux/.local/lib/python3.10/site-packages/accelerate/accelerator.py", line 2157, in clip_grad_norm_
self.unscale_gradients()
File "/home/santi-linux/.local/lib/python3.10/site-packages/accelerate/accelerator.py", line 2107, in unscale_gradients
self.scaler.unscale_(opt)
File "/home/santi-linux/.local/lib/python3.10/site-packages/torch/amp/grad_scaler.py", line 337, in unscale_
optimizer_state["found_inf_per_device"] = self._unscale_grads_(
File "/home/santi-linux/.local/lib/python3.10/site-packages/torch/amp/grad_scaler.py", line 259, in _unscale_grads_
raise ValueError("Attempting to unscale FP16 gradients.")
ValueError: Attempting to unscale FP16 gradients.
Traceback (most recent call last):
File "/home/santi-linux/trainer_kohya_ss/sd-scripts/sdxl_train_network.py", line 185, in
trainer.train(args)
File "/home/santi-linux/trainer_kohya_ss/sd-scripts/train_network.py", line 907, in train
accelerator.clip_grad_norm_(params_to_clip, args.max_grad_norm)
File "/home/santi-linux/.local/lib/python3.10/site-packages/accelerate/accelerator.py", line 2157, in clip_grad_norm_
self.unscale_gradients()
File "/home/santi-linux/.local/lib/python3.10/site-packages/accelerate/accelerator.py", line 2107, in unscale_gradients
self.scaler.unscale_(opt)
File "/home/santi-linux/.local/lib/python3.10/site-packages/torch/amp/grad_scaler.py", line 337, in unscale_
optimizer_state["found_inf_per_device"] = self._unscale_grads_(
File "/home/santi-linux/.local/lib/python3.10/site-packages/torch/amp/grad_scaler.py", line 259, in _unscale_grads_
raise ValueError("Attempting to unscale FP16 gradients.")
ValueError: Attempting to unscale FP16 gradients.

Traceback (most recent call last):
File "/home/santi-linux/.local/bin/accelerate", line 8, in
sys.exit(main())
File "/home/santi-linux/.local/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 46, in main
args.func(args)
File "/home/santi-linux/.local/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1075, in launch_command
simple_launcher(args)
File "/home/santi-linux/.local/lib/python3.10/site-packages/accelerate/commands/launch.py", line 681, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python3', 'sdxl_train_network.py', '--config_file=/home/santi-linux/trainer_kohya_ss/train_network_SDXL_AdamW.toml']' returned non-zero exit status 1.
Press Enter to continue...
```

### Machine and specs
Linux Ubuntu 22.04.4 LTS
Package: rocm-libs
Version: 6.0.2.60002-115~22.04
kernel 6.5.0-28-generic
AMD® Ryzen 7 5700g with radeon graphics × 16
AMD® Radeon graphics / AMD® Radeon rx 6700 xt

### Additional comments
I've been training different types of LoRAs with no issue, including lyco algos and DoRA. But I've tried to find a solution for this issue in DyLoRA and seems impossible, even downgrading or upgrading package/libraries versions like transformers, accelerate or diffusers didn't worked at all.

Start by reproducing the SDXL DyLoRA run from the reported train_network_SDXL_AdamW.toml configuration, then inspect networks/dylora and train_network.py at line 907, where accelerator.clip_grad_norm_ triggers the traceback. Compare the failing DyLoRA setup with the working LoRA, LyCORIS, or DoRA runs. Done means training passes the initial gradient-clipping step without the FP16 unscaling error.

Written by the indexing model from the issue text.