checkpoint loading problem

### Your question

using api json
in one workflow change checkpoint model and run is fast,.
but change to another workflow(same checkpointloader id but different model)get slow checkpoint loading speed
i think they do the same thing(load a new checkpoint model) but speed can be 10 times slow
most of the time waste on "model_type EPS" step

sys info
Platform: Linux
Python version: 3.10.8
pytorch version: 2.1.2+cu121
xformers version: 0.0.23.post1
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce RTX 3090 : cudaMallocAsync

### Logs

```powershell
after 1 time running and change checkpoint model to run
got prompt
Requested to load SDXL
Requested to load ControlNet
Loading 2 new models
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [00:09<00:00, 1.60it/s]
# 😺dzNodes: LayerStyle -> SaveImagePlus -> Saving image to /root/autodl-tmp/ComfyUI/output/img2style_2024-07-31_10-49-31.jpg
Prompt executed in 15.89 seconds

change workflow and different checkpoint model
model_type EPS
Using xformers attention in VAE
Using xformers attention in VAE
Requested to load SDXLClipModel
Loading 1 new model
INFO: InsightFace detection resolution lowered to (576, 576).
INFO: InsightFace detection resolution lowered to (576, 576).
Requested to load AutoencoderKL
Loading 1 new model
# 😺dzNodes: LayerStyle -> Processed 1 mask(s).
Requested to load SDXL
Loading 1 new model
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:05<00:00, 2.12it/s]
Prompt executed in 175.48 seconds
```

### Other

_No response_