[Bug]: High VRAM cards still slower and moving models due to Lora networks

### Checklist

- [x] The issue exists after disabling all extensions
- [ ] The issue exists on a clean installation of webui
- [ ] The issue is caused by an extension, but I believe it is caused by a bug in the webui
- [x] The issue exists in the current version of the webui
- [x] The issue has not been reported before recently
- [ ] The issue has been reported before but has not been fixed yet

### What happened?

Initially for me the main a1111 was faster on a 3090, so I found that it as an issue with moving models, and when using the command line --always-gpu it made my gens 5 seconds faster on a 3090.
I read that the issue of moving models was fixed, but when using the latest Forge version it's still moving loras and 2+ seconds slower in comparison than with the command on. The "Number of Lora networks to keep cached in memory" setting is active, so I'm guessing it's storing it in the RAM and moving to VRAM.
I would just use the command, but the problem is that when I change full checkpoints it doesn't unload the previous one, forcing me to restart the WebUI when changing checkpoints 2-3 times.

If this isn't just a bug my solution would be adding a checkbox to this setting that says "Keep lora networks in VRAM" or maybe a setting like --always-gpu-no-checkpoint that keeps every model except for checkpoints in VRAM for users with high VRAM cards.

### Steps to reproduce the problem

1. text to image using an SDXL lora
2. do it a few times with the same lora
3. notice it's always taking 2-4 seconds to move models
4. activate --always-gpu
5. text to image same prompt
6. notice after the first try it doesn't move lora anymore and is faster

### What should have happened?

It should have kept a number of lora networks in the vram that matches the "Number of Lora networks to keep cached in memory" setting

### What browsers do you use to access the UI ?

Brave

### Sysinfo

[sysinfo-.json](https://github.com/lllyasviel/stable-diffusion-webui-forge/files/14292629/sysinfo-.json)

### Console logs

```Shell
venv "\Forge\stable-diffusion-webui-forge\venv\Scripts\Python.exe"
Python 3.10.9 () [MSC v.1934 64 bit (AMD64)]
Version: f0.0.12-latest-155-gd81e353d
Commit hash: d81e353d8928147bbd973068d0efbb2802affe0f
loading WD14-tagger reqs from \Forge\stable-diffusion-webui-forge\extensions\stable-diffusion-webui-wd14-tagger\requirements.txt
Checking WD14-tagger requirements.
Launching Web UI with arguments:
Total VRAM 24576 MB, total RAM 65446 MB
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce RTX 3090 : native
VAE dtype: torch.bfloat16
I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
WARNING:tensorflow:\Forge\stable-diffusion-webui-forge\venv\lib\site-packages\keras\src\losses.py:2976: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

no module 'xformers'. Processing without...
no module 'xformers'. Processing without...
No module 'xformers'. Proceeding without it.
\stable-diffusion-webui-forge\venv\lib\site-packages\pytorch_lightning\utilities\distributed.py:258: LightningDeprecationWarning: `pytorch_lightning.utilities.distributed.rank_zero_only` has been deprecated in v1.8.1 and will be removed in v2.0.0. You can import it from `pytorch_lightning.utilities` instead.
rank_zero_deprecation(
Using pytorch cross attention
ControlNet preprocessor location: \stable-diffusion-webui-forge\models\ControlNetPreprocessor
Tag Autocomplete: Could not locate model-keyword extension, Lora trigger word completion will be limited to those added through the extra networks menu.
[-] ADetailer initialized. version: 24.1.2, num models: 14
== WD14 tagger /gpu:0, uname_result(system='Windows', node='DESKTOP-39VISVT', release='10', version='10.0.19045', machine='AMD64') ==
Loading weights [821aa5537f] from
- ControlNet - INFO - ControlNet UI callback registered.
model_type EPS
UNet ADM Dimension 2816
Using pytorch attention in VAE
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
Using pytorch attention in VAE
extra {'cond_stage_model.clip_l.logit_scale', 'cond_stage_model.clip_l.text_projection', 'cond_stage_model.clip_g.transformer.text_model.embeddings.position_ids'}
Loading VAE weights specified in settings:
To load target model SDXLClipModel
Begin to load 1 model
Running on local URL: http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 25.7s (prepare environment: 5.7s, import torch: 9.3s, initialize shared: 0.1s, other imports: 0.4s, list SD models: 0.5s, load scripts: 4.6s, create ui: 4.3s, gradio launch: 0.6s, app_started_callback: 0.1s).
Moving model(s) has taken 2.10 seconds
Model loaded in 8.5s (load weights from disk: 1.0s, forge load real models: 2.2s, load VAE: 0.1s, calculate empty prompt: 5.2s).
To load target model SDXLClipModel
Begin to load 1 model
unload clone 0
Moving model(s) has taken 2.09 seconds
To load target model SDXL
Begin to load 1 model
Moving model(s) has taken 2.73 seconds
100%|██████████████████████████████████████████████████████████████████████████████████| 28/28 [00:09<00:00, 2.83it/s]
To load target model AutoencoderKL█████████████████████████████████████████████████████| 28/28 [00:09<00:00, 2.99it/s]
Begin to load 1 model
Total progress: 100%|██████████████████████████████████████████████████████████████████| 28/28 [00:10<00:00, 2.72it/s]
To load target model SDXL██████████████████████████████████████████████████████████████| 28/28 [00:10<00:00, 2.99it/s]
Begin to load 1 model
unload clone 1
Moving model(s) has taken 2.72 seconds
100%|██████████████████████████████████████████████████████████████████████████████████| 28/28 [00:09<00:00, 2.97it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 28/28 [00:10<00:00, 2.80it/s]
To load target model SDXL██████████████████████████████████████████████████████████████| 28/28 [00:10<00:00, 2.97it/s]
Begin to load 1 model
unload clone 1
Moving model(s) has taken 2.78 seconds
100%|██████████████████████████████████████████████████████████████████████████████████| 28/28 [00:10<00:00, 2.78it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 28/28 [00:10<00:00, 2.65it/s]
To load target model SDXL██████████████████████████████████████████████████████████████| 28/28 [00:10<00:00, 2.63it/s]
Begin to load 1 model
unload clone 1
Moving model(s) has taken 2.79 seconds
100%|██████████████████████████████████████████████████████████████████████████████████| 28/28 [00:08<00:00, 3.31it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 28/28 [00:09<00:00, 3.09it/s]
To load target model SDXL██████████████████████████████████████████████████████████████| 28/28 [00:09<00:00, 3.28it/s]
Begin to load 1 model
unload clone 1
Moving model(s) has taken 2.54 seconds
100%|██████████████████████████████████████████████████████████████████████████████████| 28/28 [00:08<00:00, 3.26it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 28/28 [00:09<00:00, 3.06it/s]
To load target model SDXLClipModel█████████████████████████████████████████████████████| 28/28 [00:09<00:00, 3.30it/s]
Begin to load 1 model
```

### Additional information

This bug might be happening because I have two gpus, not sure if everyone is having this issue. The gpu being used is the 3090 though, as you can see in the console.

Start by reproducing the SDXL LoRA workflow with the “Number of Lora networks to keep cached in memory” setting, then compare its console movement timings with and without --always-gpu. Check whether repeated generations move cached LoRAs between RAM and VRAM, and verify that changing checkpoints still unloads the previous checkpoint. Done means the configured LoRA cache remains in VRAM without breaking checkpoint switching.

Written by the indexing model from the issue text.