[Bug]: Initial lag / slowdown when switching between Checkpoints and LoRAs

### Checklist

- [X] The issue exists after disabling all extensions
- [X] The issue exists on a clean installation of webui
- [ ] The issue is caused by an extension, but I believe it is caused by a bug in the webui
- [X] The issue exists in the current version of the webui
- [ ] The issue has not been reported before recently
- [ ] The issue has been reported before but has not been fixed yet

### What happened?

I have found that Forge is considerably slower and "laggier" when switching between Checkpoints, LoRAs, and the other extra networks tabs.

Compared to A1111, there's a long loading time on a fresh start of the program, but it seems that A1111 handles the content caching differently.

With Forge, if you switch between Checkpoints and LoRA's for example, there's then a period where nothing in the UI is clickable. I can't select a model, switch back to Checkpoints.

Going from Checkpoints to LoRA to Checkpoints again takes ~30 seconds for me.
With A1111, it takes 5 seconds.

This won't be noticable until you have a large amount of models.
I have ~1000 checkpoints and ~10.000 LoRAs.

### Steps to reproduce the problem

1. Download a 1000 checkpoints and 10.000 LoRAs.
2. Enter the Checkpoints sub-tab.
3. Wait a minute for it to load.
4. Enter LoRA sub-tab.
5. Wait another minute for it to load.
6. Switch back to Checkpoints sub-tab.
7. Wait ~10 seconds for it to load.
8. Switch back to LoRA sub-tab
9. Wait ~10 seconds for it to load.

And for 3-5 seconds after switching to another tab, the UI is unresponsive.
I can still scroll the mouse-wheel to scroll the models list, but nothing anywhere can be clicked. The mouse cursor does not turn into a link-cursor (hand).

Additionally, even typing in the filter box is incredibly slow. It's like it re-filters everything and reloads it for every key that you type. I have to wait 5 seconds before each letter appears, if I type into the LoRA filter box with my 11k models.

### What should have happened?

Forge should ideally perform on par with A1111, or better.

### What browsers do you use to access the UI ?

Mozilla Firefox

### Sysinfo

[sysinfo-2024-02-12-21-58.json](https://github.com/lllyasviel/stable-diffusion-webui-forge/files/14251319/sysinfo-2024-02-12-21-58.json)

### Console logs

```Shell
venv "C:\AI\stable-diffusion-webui-forge\venv\Scripts\Python.exe"
Python 3.10.9 (tags/v3.10.9:1dd9be6, Dec 6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)]
Version: f0.0.12-latest-113-ge11753ff
Commit hash: e11753ff844b6e06529287f65b2f9efd1fd76cc6
no module 'xformers'. Processing without...
no module 'xformers'. Processing without...
No module 'xformers'. Proceeding without it.
Total VRAM 8192 MB, total RAM 65270 MB
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce RTX 3070 : native
VAE dtype: torch.bfloat16
Installing requirements for Face Editor
Faceswaplab : Use GPU requirements
Checking faceswaplab requirements
0.007527400040999055
Launching Web UI with arguments: --port 7861
Total VRAM 8192 MB, total RAM 65270 MB
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce RTX 3070 : native
VAE dtype: torch.bfloat16
Using pytorch cross attention
ControlNet preprocessor location: C:\AI\stable-diffusion-webui-forge\models\ControlNetPreprocessor
Civitai Helper: Get Custom Model Folder
Tag Autocomplete: Could not locate model-keyword extension, Lora trigger word completion will be limited to those added through the extra networks menu.
[-] ADetailer initialized. version: 24.1.2, num models: 58
23:00:12 - ReActor - STATUS - Running v0.6.1 on Device: CPU
Thumbnailizer initialized
Loading weights [f7a1beed86] from C:\AI\stable-diffusion-webui-forge\models\Stable-diffusion\Checkpoints\10 - SDXL\colossusProjectXLSFW_v53Trained.safetensors
2024-02-12 23:00:13,482 - ControlNet - INFO - ControlNet UI callback registered.
model_type EPS
UNet ADM Dimension 2816
Civitai Helper: Set Proxy:
Using pytorch attention in VAE
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
Using pytorch attention in VAE
extra {'cond_stage_model.clip_l.logit_scale', 'cond_stage_model.clip_l.text_projection', 'cond_stage_model.clip_g.transformer.text_model.embeddings.position_ids'}
left over keys: dict_keys(['conditioner.embedders.0.logit_scale', 'conditioner.embedders.0.text_projection', 'conditioner.embedders.1.model.transformer.text_model.embeddings.position_ids', 'decoder.conv_in.bias', 'decoder.conv_in.weight', 'decoder.conv_out.bias', 'decoder.conv_out.weight', 'decoder.mid.attn_1.k.bias', 'decoder.mid.attn_1.k.weight', 'decoder.mid.attn_1.norm.bias', 'decoder.mid.attn_1.norm.weight', 'decoder.mid.attn_1.proj_out.bias', 'decoder.mid.attn_1.proj_out.weight', 'decoder.mid.attn_1.q.bias', 'decoder.mid.attn_1.q.weight', 'decoder.mid.attn_1.v.bias', 'decoder.mid.attn_1.v.weight', 'decoder.mid.block_1.conv1.bias', 'decoder.mid.block_1.conv1.weight', 'decoder.mid.block_1.conv2.bias', 'decoder.mid.block_1.conv2.weight', 'decoder.mid.block_1.norm1.bias', 'decoder.mid.block_1.norm1.weight', 'decoder.mid.block_1.norm2.bias', 'decoder.mid.block_1.norm2.weight', 'decoder.mid.block_2.conv1.bias', 'decoder.mid.block_2.conv1.weight', 'decoder.mid.block_2.conv2.bias', 'decoder.mid.block_2.conv2.weight', 'decoder.mid.block_2.norm1.bias', 'decoder.mid.block_2.norm1.weight', 'decoder.mid.block_2.norm2.bias', 'decoder.mid.block_2.norm2.weight', 'decoder.norm_out.bias', 'decoder.norm_out.weight', 'decoder.up.0.block.0.conv1.bias', 'decoder.up.0.block.0.conv1.weight', 'decoder.up.0.block.0.conv2.bias', 'decoder.up.0.block.0.conv2.weight', 'decoder.up.0.block.0.nin_shortcut.bias', 'decoder.up.0.block.0.nin_shortcut.weight', 'decoder.up.0.block.0.norm1.bias', 'decoder.up.0.block.0.norm1.weight', 'decoder.up.0.block.0.norm2.bias', 'decoder.up.0.block.0.norm2.weight', 'decoder.up.0.block.1.conv1.bias', 'decoder.up.0.block.1.conv1.weight', 'decoder.up.0.block.1.conv2.bias', 'decoder.up.0.block.1.conv2.weight', 'decoder.up.0.block.1.norm1.bias', 'decoder.up.0.block.1.norm1.weight', 'decoder.up.0.block.1.norm2.bias', 'decoder.up.0.block.1.norm2.weight', 'decoder.up.0.block.2.conv1.bias', 'decoder.up.0.block.2.conv1.weight', 'decoder.up.0.block.2.conv2.bias', 'decoder.up.0.block.2.conv2.weight', 'decoder.up.0.block.2.norm1.bias', 'decoder.up.0.block.2.norm1.weight', 'decoder.up.0.block.2.norm2.bias', 'decoder.up.0.block.2.norm2.weight', 'decoder.up.1.block.0.conv1.bias', 'decoder.up.1.block.0.conv1.weight', 'decoder.up.1.block.0.conv2.bias', 'decoder.up.1.block.0.conv2.weight', 'decoder.up.1.block.0.nin_shortcut.bias', 'decoder.up.1.block.0.nin_shortcut.weight', 'decoder.up.1.block.0.norm1.bias', 'decoder.up.1.block.0.norm1.weight', 'decoder.up.1.block.0.norm2.bias', 'decoder.up.1.block.0.norm2.weight', 'decoder.up.1.block.1.conv1.bias', 'decoder.up.1.block.1.conv1.weight', 'decoder.up.1.block.1.conv2.bias', 'decoder.up.1.block.1.conv2.weight', 'decoder.up.1.block.1.norm1.bias', 'decoder.up.1.block.1.norm1.weight', 'decoder.up.1.block.1.norm2.bias', 'decoder.up.1.block.1.norm2.weight', 'decoder.up.1.block.2.conv1.bias', 'decoder.up.1.block.2.conv1.weight', 'decoder.up.1.block.2.conv2.bias', 'decoder.up.1.block.2.conv2.weight', 'decoder.up.1.block.2.norm1.bias', 'decoder.up.1.block.2.norm1.weight', 'decoder.up.1.block.2.norm2.bias', 'decoder.up.1.block.2.norm2.weight', 'decoder.up.1.upsample.conv.bias', 'decoder.up.1.upsample.conv.weight', 'decoder.up.2.block.0.conv1.bias', 'decoder.up.2.block.0.conv1.weight', 'decoder.up.2.block.0.conv2.bias', 'decoder.up.2.block.0.conv2.weight', 'decoder.up.2.block.0.norm1.bias', 'decoder.up.2.block.0.norm1.weight', 'decoder.up.2.block.0.norm2.bias', 'decoder.up.2.block.0.norm2.weight', 'decoder.up.2.block.1.conv1.bias', 'decoder.up.2.block.1.conv1.weight', 'decoder.up.2.block.1.conv2.bias', 'decoder.up.2.block.1.conv2.weight', 'decoder.up.2.block.1.norm1.bias', 'decoder.up.2.block.1.norm1.weight', 'decoder.up.2.block.1.norm2.bias', 'decoder.up.2.block.1.norm2.weight', 'decoder.up.2.block.2.conv1.bias', 'decoder.up.2.block.2.conv1.weight', 'decoder.up.2.block.2.conv2.bias', 'decoder.up.2.block.2.conv2.weight', 'decoder.up.2.block.2.norm1.bias', 'decoder.up.2.block.2.norm1.weight', 'decoder.up.2.block.2.norm2.bias', 'decoder.up.2.block.2.norm2.weight', 'decoder.up.2.upsample.conv.bias', 'decoder.up.2.upsample.conv.weight', 'decoder.up.3.block.0.conv1.bias', 'decoder.up.3.block.0.conv1.weight', 'decoder.up.3.block.0.conv2.bias', 'decoder.up.3.block.0.conv2.weight', 'decoder.up.3.block.0.norm1.bias', 'decoder.up.3.block.0.norm1.weight', 'decoder.up.3.block.0.norm2.bias', 'decoder.up.3.block.0.norm2.weight', 'decoder.up.3.block.1.conv1.bias', 'decoder.up.3.block.1.conv1.weight', 'decoder.up.3.block.1.conv2.bias', 'decoder.up.3.block.1.conv2.weight', 'decoder.up.3.block.1.norm1.bias', 'decoder.up.3.block.1.norm1.weight', 'decoder.up.3.block.1.norm2.bias', 'decoder.up.3.block.1.norm2.weight', 'decoder.up.3.block.2.conv1.bias', 'decoder.up.3.block.2.conv1.weight', 'decoder.up.3.block.2.conv2.bias', 'decoder.up.3.block.2.conv2.weight', 'decoder.up.3.block.2.norm1.bias', 'decoder.up.3.block.2.norm1.weight', 'decoder.up.3.block.2.norm2.bias', 'decoder.up.3.block.2.norm2.weight', 'decoder.up.3.upsample.conv.bias', 'decoder.up.3.upsample.conv.weight', 'encoder.conv_in.bias', 'encoder.conv_in.weight', 'encoder.conv_out.bias', 'encoder.conv_out.weight', 'encoder.down.0.block.0.conv1.bias', 'encoder.down.0.block.0.conv1.weight', 'encoder.down.0.block.0.conv2.bias', 'encoder.down.0.block.0.conv2.weight', 'encoder.down.0.block.0.norm1.bias', 'encoder.down.0.block.0.norm1.weight', 'encoder.down.0.block.0.norm2.bias', 'encoder.down.0.block.0.norm2.weight', 'encoder.down.0.block.1.conv1.bias', 'encoder.down.0.block.1.conv1.weight', 'encoder.down.0.block.1.conv2.bias', 'encoder.down.0.block.1.conv2.weight', 'encoder.down.0.block.1.norm1.bias', 'encoder.down.0.block.1.norm1.weight', 'encoder.down.0.block.1.norm2.bias', 'encoder.down.0.block.1.norm2.weight', 'encoder.down.0.downsample.conv.bias', 'encoder.down.0.downsample.conv.weight', 'encoder.down.1.block.0.conv1.bias', 'encoder.down.1.block.0.conv1.weight', 'encoder.down.1.block.0.conv2.bias', 'encoder.down.1.block.0.conv2.weight', 'encoder.down.1.block.0.nin_shortcut.bias', 'encoder.down.1.block.0.nin_shortcut.weight', 'encoder.down.1.block.0.norm1.bias', 'encoder.down.1.block.0.norm1.weight', 'encoder.down.1.block.0.norm2.bias', 'encoder.down.1.block.0.norm2.weight', 'encoder.down.1.block.1.conv1.bias', 'encoder.down.1.block.1.conv1.weight', 'encoder.down.1.block.1.conv2.bias', 'encoder.down.1.block.1.conv2.weight', 'encoder.down.1.block.1.norm1.bias', 'encoder.down.1.block.1.norm1.weight', 'encoder.down.1.block.1.norm2.bias', 'encoder.down.1.block.1.norm2.weight', 'encoder.down.1.downsample.conv.bias', 'encoder.down.1.downsample.conv.weight', 'encoder.down.2.block.0.conv1.bias', 'encoder.down.2.block.0.conv1.weight', 'encoder.down.2.block.0.conv2.bias', 'encoder.down.2.block.0.conv2.weight', 'encoder.down.2.block.0.nin_shortcut.bias', 'encoder.down.2.block.0.nin_shortcut.weight', 'encoder.down.2.block.0.norm1.bias', 'encoder.down.2.block.0.norm1.weight', 'encoder.down.2.block.0.norm2.bias', 'encoder.down.2.block.0.norm2.weight', 'encoder.down.2.block.1.conv1.bias', 'encoder.down.2.block.1.conv1.weight', 'encoder.down.2.block.1.conv2.bias', 'encoder.down.2.block.1.conv2.weight', 'encoder.down.2.block.1.norm1.bias', 'encoder.down.2.block.1.norm1.weight', 'encoder.down.2.block.1.norm2.bias', 'encoder.down.2.block.1.norm2.weight', 'encoder.down.2.downsample.conv.bias', 'encoder.down.2.downsample.conv.weight', 'encoder.down.3.block.0.conv1.bias', 'encoder.down.3.block.0.conv1.weight', 'encoder.down.3.block.0.conv2.bias', 'encoder.down.3.block.0.conv2.weight', 'encoder.down.3.block.0.norm1.bias', 'encoder.down.3.block.0.norm1.weight', 'encoder.down.3.block.0.norm2.bias', 'encoder.down.3.block.0.norm2.weight', 'encoder.down.3.block.1.conv1.bias', 'encoder.down.3.block.1.conv1.weight', 'encoder.down.3.block.1.conv2.bias', 'encoder.down.3.block.1.conv2.weight', 'encoder.down.3.block.1.norm1.bias', 'encoder.down.3.block.1.norm1.weight', 'encoder.down.3.block.1.norm2.bias', 'encoder.down.3.block.1.norm2.weight', 'encoder.mid.attn_1.k.bias', 'encoder.mid.attn_1.k.weight', 'encoder.mid.attn_1.norm.bias', 'encoder.mid.attn_1.norm.weight', 'encoder.mid.attn_1.proj_out.bias', 'encoder.mid.attn_1.proj_out.weight', 'encoder.mid.attn_1.q.bias', 'encoder.mid.attn_1.q.weight', 'encoder.mid.attn_1.v.bias', 'encoder.mid.attn_1.v.weight', 'encoder.mid.block_1.conv1.bias', 'encoder.mid.block_1.conv1.weight', 'encoder.mid.block_1.conv2.bias', 'encoder.mid.block_1.conv2.weight', 'encoder.mid.block_1.norm1.bias', 'encoder.mid.block_1.norm1.weight', 'encoder.mid.block_1.norm2.bias', 'encoder.mid.block_1.norm2.weight', 'encoder.mid.block_2.conv1.bias', 'encoder.mid.block_2.conv1.weight', 'encoder.mid.block_2.conv2.bias', 'encoder.mid.block_2.conv2.weight', 'encoder.mid.block_2.norm1.bias', 'encoder.mid.block_2.norm1.weight', 'encoder.mid.block_2.norm2.bias', 'encoder.mid.block_2.norm2.weight', 'encoder.norm_out.bias', 'encoder.norm_out.weight', 'post_quant_conv.bias', 'post_quant_conv.weight', 'quant_conv.bias', 'quant_conv.weight'])
Loading VAE weights specified in settings: C:\AI\stable-diffusion-webui-forge\models\VAE\sdxl_vae.safetensors
Running on local URL: http://127.0.0.1:7861
No Image data blocks found.

To create a public link, set `share=True` in `launch()`.
Startup time: 33.0s (prepare environment: 12.2s, import torch: 2.6s, import gradio: 0.7s, setup paths: 0.5s, other imports: 0.7s, list SD models: 3.1s, load scripts: 6.6s, create ui: 6.1s, gradio launch: 0.2s).
No Image data blocks found.
No Image data blocks found.
To load target model SDXLClipModel
Begin to load 1 model
Moving model(s) has taken 0.48 seconds
Model loaded in 8.4s (load weights from disk: 0.1s, forge load real models: 4.3s, forge finalize: 0.4s, load VAE: 0.8s, load textual inversion embeddings: 1.9s, calculate empty prompt: 0.8s).
```

### Additional information

Both A1111 and Forge are located on an NVMe drive.
I have tried both in regular mode and incognito mode in Chrome and Firefox, the issue is there for all cases.
I have disabled ad-blockers and the issue is still there.

Start by reproducing the lag while switching the Checkpoints and LoRA extra-networks tabs and typing in their filter boxes with the reported large model collections. Trace the tab-loading, caching, and filtering entry points responsible for the unresponsive UI; done means switching and filtering no longer incur the reported multi-second delays.

Written by the indexing model from the issue text.