[Bug]: Very long moving models time when using Lora/Lycoris

### Checklist

- [X] The issue exists after disabling all extensions
- [X] The issue exists on a clean installation of webui
- [ ] The issue is caused by an extension, but I believe it is caused by a bug in the webui
- [X] The issue exists in the current version of the webui
- [X] The issue has not been reported before recently
- [ ] The issue has been reported before but has not been fixed yet

### What happened?

I didn't find any way to fix this problem, and i spend a lot of time just tweaking certain settings in webui to figure out how to reduce time of moving models

### Steps to reproduce the problem

1. Instal forge on completely new Sata SSD with a lot of free space
2. Launch forge
3. Choose any Lora/Lycoris
4. Press "Generate"
5. Hope that you somehow got this issue

### What should have happened?

When i was usng A1111 webui i didn't have any problems, Lora's just work without any additional loading times, my pc not very powerful, but when i tried to research this issue i found people that have similar troubles with checkpoints

### What browsers do you use to access the UI ?

Mozilla Firefox

### Sysinfo

[sysinfo-2024-04-26-21-29.json](https://github.com/lllyasviel/stable-diffusion-webui-forge/files/15135453/sysinfo-2024-04-26-21-29.json)

### Console logs

```Shell
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Version: f0.0.17v1.8.0rc-latest-276-g29be1da7
Commit hash: 29be1da7cf2b5dccfc70fbdd33eb35c56a31ffb7
Launching Web UI with arguments: --cuda-stream
Total VRAM 6144 MB, total RAM 12244 MB
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce GTX 1660 SUPER : native
VAE dtype: torch.float32
CUDA Stream Activated: True
Using pytorch cross attention
*** "Disable all extensions" option was set, will only load built-in extensions ***
ControlNet preprocessor location: G:\SD\webui\models\ControlNetPreprocessor
Loading weights [f1f226aa36] from G:\SD\webui\models\Stable-diffusion\Anime Lightning - AutismMix4steps.safetensors
2024-04-26 22:56:03,675 - ControlNet - INFO - ControlNet UI callback registered.
Running on local URL: http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 78.3s (initial startup: 0.3s, prepare environment: 22.0s, import torch: 20.0s, import gradio: 3.4s, setup paths: 3.2s, initialize shared: 0.4s, other imports: 4.6s, list SD models: 0.6s, load scripts: 6.0s, create ui: 9.5s, gradio launch: 8.0s).
model_type EPS
UNet ADM Dimension 2816
Using pytorch attention in VAE
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
Using pytorch attention in VAE
extra {'cond_stage_model.clip_l.text_projection', 'cond_stage_model.clip_g.transformer.text_model.embeddings.position_ids', 'cond_stage_model.clip_l.logit_scale'}
To load target model SDXLClipModel
Begin to load 1 model
Moving model(s) has taken 0.50 seconds
Model loaded in 1218.7s (calculate hash: 0.2s, load weights from disk: 756.5s, forge solving config: 7.0s, forge instantiate config: 17.4s, forge load real models: 395.5s, forge set components: 0.4s, forge finalize: 3.3s, load VAE: 0.3s, load textual inversion embeddings: 1.7s, scripts callbacks: 0.2s, calculate empty prompt: 36.2s).
[LORA] Loaded G:\SD\webui\models\Lora\Characters\Ilulu_Sdxl-10.safetensors for SDXL-UNet with 722 keys at weight 1.0 (skipped 0 keys)
[LORA] Loaded G:\SD\webui\models\Lora\Characters\Ilulu_Sdxl-10.safetensors for SDXL-CLIP with 264 keys at weight 1.0 (skipped 0 keys)
To load target model SDXLClipModel
Begin to load 1 model
Reuse 1 loaded models
Moving model(s) has taken 7.34 seconds
To load target model SDXL
Begin to load 1 model
[Memory Management] Current Free GPU Memory (MB) = 4056.43994140625
[Memory Management] Model Memory (MB) = 4897.086494445801
[Memory Management] Minimal Inference Memory (MB) = 1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) = -1864.6465530395508
[Memory Management] Requested ASYNC Preserved Memory (MB) = 2332.6461086273193
[Memory Management] Parameters Loaded to ASYNC Stream (MB) = 2564.74853515625
[Memory Management] Parameters Loaded to GPU (MB) = 2332.2998123168945
Moving model(s) has taken 451.26 seconds
100%|████████████████████████████████████████████████████████████████████████████████████| 6/6 [03:25<00:00, 34.22s/it]
To load target model AutoencoderKL███████████████████████████████████████████████████████| 6/6 [01:44<00:00, 17.59s/it]

Start by reproducing the reported delay with a clean Forge installation, an SDXL checkpoint, and the listed LoRA/Lycoris model, using the supplied console log and sysinfo JSON for comparison. Measure the model-moving time before and after applying the LoRA, then trace the model-loading path responsible for that transfer. Done means the added model no longer causes the reported excessive delay, with a regression test or comparable timing evidence.

Written by the indexing model from the issue text.