Lots of errors with LoRAs made with The Last Ben's Runpod SDXL training template

The LoRAs in question were made with The Last Ben's Runpod template for SDXL.

I'm having a problem, where the Colab with LoRAs give always errors like this, regardless of the rank:

```
ERROR diffusion_model.input_blocks.4.1.proj_out.weight shape '[768, 768]' is invalid for input of size 409600
ERROR diffusion_model.input_blocks.4.1.proj_in.weight shape '[768, 768]' is invalid for input of size 409600
ERROR diffusion_model.input_blocks.4.1.transformer_blocks.0.attn2.to_out.0.weight shape '[768, 768]' is invalid for input of size 409600
ERROR diffusion_model.input_blocks.4.1.transformer_blocks.0.ff.net.0.proj.weight shape '[6144, 768]' is invalid for input of size 3276800
ERROR diffusion_model.input_blocks.4.1.transformer_blocks.0.attn1.to_out.0.weight shape '[768, 768]' is invalid for input of size 409600
ERROR diffusion_model.input_blocks.4.1.transformer_blocks.0.attn1.to_q.weight shape '[768, 768]' is invalid for input of size 409600
ERROR diffusion_model.input_blocks.4.1.transformer_blocks.0.ff.net.2.weight shape '[768, 3072]' is invalid for input of size 1638400
ERROR diffusion_model.input_blocks.4.1.transformer_blocks.0.attn2.to_k.weight shape '[768, 1280]' is invalid for input of size 1310720
ERROR diffusion_model.input_blocks.4.1.transformer_blocks.0.attn1.to_k.weight shape '[768, 768]' is invalid for input of size 409600
ERROR diffusion_model.input_blocks.4.1.transformer_blocks.0.attn2.to_q.weight shape '[768, 768]' is invalid for input of size 409600
ERROR diffusion_model.input_blocks.4.1.transformer_blocks.0.attn2.to_v.weight shape '[768, 1280]' is invalid for input of size 1310720
ERROR diffusion_model.input_blocks.4.1.transformer_blocks.0.attn1.to_v.weight shape '[768, 768]' is invalid for input of size 409600
ERROR diffusion_model.input_blocks.4.1.transformer_blocks.1.attn2.to_out.0.weight shape '[768, 768]' is invalid for input of size 409600
ERROR diffusion_model.input_blocks.4.1.transformer_blocks.1.ff.net.0.proj.weight shape '[6144, 768]' is invalid for input of size 3276800
ERROR diffusion_model.input_blocks.4.1.transformer_blocks.1.attn1.to_out.0.weight shape '[768, 768]' is invalid for input of size 409600
ERROR diffusion_model.input_blocks.4.1.transformer_blocks.1.attn1.to_q.weight shape '[768, 768]' is invalid for input of size 409600
ERROR diffusion_model.input_blocks.4.1.transformer_blocks.1.ff.net.2.weight shape '[768, 3072]' is invalid for input of size 1638400
ERROR diffusion_model.input_blocks.4.1.transformer_blocks.1.attn2.to_k.weight shape '[768, 1280]' is invalid for input of size 1310720
ERROR diffusion_model.input_blocks.4.1.transformer_blocks.1.attn1.to_k.weight shape '[768, 768]' is invalid for input of size 409600
ERROR diffusion_model.input_blocks.4.1.transformer_blocks.1.attn2.to_q.weight shape '[768, 768]' is invalid for input of size 409600
ERROR diffusion_model.input_blocks.4.1.transformer_blocks.1.attn2.to_v.weight shape '[768, 1280]' is invalid for input of size 1310720
ERROR diffusion_model.input_blocks.4.1.transformer_blocks.1.attn1.to_v.weight shape '[768, 768]' is invalid for input of size 409600
ERROR diffusion_model.input_blocks.5.1.proj_out.weight shape '[768, 768]' is invalid for input of size 409600
ERROR diffusion_model.input_blocks.5.1.proj_in.weight shape '[768, 768]' is invalid for input of size 409600
```

This is just an excerpt, there's tons more for every render. The LoRAs seem to work though.

I'm getting them also for middle and output blocks, whatever that means: (again, a short excerpt)
```
ERROR diffusion_model.middle_block.1.transformer_blocks.3.attn1.to_v.weight shape '[1536, 1536]' is invalid for input of size 1638400
ERROR diffusion_model.output_blocks.3.1.proj_out.weight shape '[1536, 1536]' is invalid for input of size 409600
ERROR diffusion_model.output_blocks.3.1.proj_in.weight shape '[1536, 1536]' is invalid for input of size 409600
ERROR diffusion_model.output_blocks.3.1.transformer_blocks.0.attn2.to_out.0.weight shape '[1536, 1536]' is invalid for input of size 409600
```

Also, just now got a lot of stuff like this too:
```
model_type EPS
adm 2560
making attention of type 'vanilla-xformers' with 512 in_channels
building MemoryEfficientAttnBlock with 512 in_channels...
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla-xformers' with 512 in_channels
building MemoryEfficientAttnBlock with 512 in_channels...
NOT LOADED diffusion_model.input_blocks.7.1.transformer_blocks.4.attn2.to_v.weight
NOT LOADED diffusion_model.input_blocks.7.1.transformer_blocks.4.attn1.to_k.weight
NOT LOADED diffusion_model.input_blocks.7.1.transformer_blocks.4.attn1.to_q.weight
NOT LOADED diffusion_model.input_blocks.7.1.transformer_blocks.4.attn1.to_out.0.weight
NOT LOADED diffusion_model.input_blocks.7.1.transformer_blocks.4.attn2.to_k.weight
NOT LOADED diffusion_model.input_blocks.7.1.transformer_blocks.4.ff.net.0.proj.weight
NOT LOADED diffusion_model.input_blocks.7.1.transformer_blocks.4.attn2.to_q.weight
NOT LOADED diffusion_model.input_blocks.7.1.transformer_blocks.4.attn2.to_out.0.weight
NOT LOADED diffusion_model.input_blocks.7.1.transformer_blocks.4.ff.net.2.weight
NOT LOADED diffusion_model.input_blocks.7.1.transformer_blocks.4.attn1.to_v.weight
NOT LOADED diffusion_model.input_blocks.7.1.transformer_blocks.5.attn2.to_v.weight
```

```
86% 12/14 [00:01<00:00, 7.87it/s]/usr/local/lib/python3.10/dist-packages/torchsde/_brownian/brownian_interval.py:585: UserWarning: Should have ta>=t0 but got ta=0.02916753850877285 and t0=0.029168.
warnings.warn(f"Should have ta>=t0 but got ta={ta} and t0={self._start}.")
```

This is with the uni_pc and dpm_2m_sde_gpu samplers.