loading models veryyyyyyy slow！

### Expected Behavior

run fast

### Actual Behavior

1. run comfyui,and run the first simple workflow,all good
2. load a complex workflow and run, seems ok
3. back to the first simple workflow, from model_type EPS waiting very long time
4. after that all workflow slow from model_type EPS step

### Steps to Reproduce

1. run comfyui,and run the first simple workflow,all good
2. change anthoer instantid workflow,its ok
3. load a complex workflow and run, little slow
4. back to the first simple workflow, from model_type EPS waiting very long time(1min)
5. after that all workflow slow from model_type EPS step

simple

instantid

complex one

### Debug Logs

```powershell
first workflow t2i(ok)
[rgthree] Using rgthree's optimized recursive execution.
[rgthree] First run patching recursive_output_delete_if_changed and recursive_will_execute.
[rgthree] Note: If execution seems broken due to forward ComfyUI changes, you can disable the optimization from rgthree settings in ComfyUI.
model_type EPS
Using xformers attention in VAE
Using xformers attention in VAE
Requested to load SD1ClipModel
Loading 1 new model
model_path is /root/autodl-tmp/ComfyUI/custom_nodes/comfyui_controlnet_aux/ckpts/lllyasviel/Annotators/ControlNetHED.pth
Requested to load AutoencoderKL
Loading 1 new model
Requested to load ControlNet
Requested to load BaseModel
Loading 2 new models
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:04<00:00, 4.45it/s]
Prompt executed in 7.94 seconds

second workflow,instantid(ok)
[rgthree] Using rgthree's optimized recursive execution.
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: /root/autodl-tmp/ComfyUI/models/insightface/models/antelopev2/1k3d68.onnx landmark_3d_68 ['None', 3, 192, 192] 0.0 1.0
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: /root/autodl-tmp/ComfyUI/models/insightface/models/antelopev2/2d106det.onnx landmark_2d_106 ['None', 3, 192, 192] 0.0 1.0
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: /root/autodl-tmp/ComfyUI/models/insightface/models/antelopev2/genderage.onnx genderage ['None', 3, 96, 96] 0.0 1.0
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: /root/autodl-tmp/ComfyUI/models/insightface/models/antelopev2/glintr100.onnx recognition ['None', 3, 112, 112] 127.5 127.5
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: /root/autodl-tmp/ComfyUI/models/insightface/models/antelopev2/scrfd_10g_bnkps.onnx detection [1, 3, '?', '?'] 127.5 128.0
set det-size: (640, 640)
model_type EPS
Using xformers attention in VAE
Using xformers attention in VAE
Requested to load SDXLClipModel
Loading 1 new model
/root/miniconda3/lib/python3.10/site-packages/insightface/utils/transform.py:68: FutureWarning: `rcond` parameter will change to the default of machine precision times ``max(M, N)`` where M and N are the input matrix dimensions.
To use the future default and silence this warning we advise to pass `rcond=None`, to keep using the old, explicitly pass `rcond=-1`.
P = np.linalg.lstsq(X_homo, Y)[0].T # Affine matrix. 3 x 4
Requested to load SDXL
Requested to load ControlNet
Loading 2 new models
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:13<00:00, 2.21it/s]
Requested to load AutoencoderKL
Loading 1 new model
Prompt executed in 31.96 seconds

third workflow complex workflow,instantid+stylelayer+XL(slow)
got prompt
[rgthree] Using rgthree's optimized recursive execution.
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
/root/miniconda3/lib/python3.10/site-packages/numba/np/ufunc/parallel.py:371: NumbaWarning: The TBB threading layer requires TBB version 2021 update 6 or later i.e., TBB_INTERFACE_VERSION >= 12060. Found TBB_INTERFACE_VERSION = 12050. The TBB threading layer is disabled.
warnings.warn(problem)
# 😺dzNodes: LayerStyle -> PersonMaskUltra V2 Processed 1 image(s).
Applied providers: ['CUDAExecutionProvider', 'CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}, 'CUDAExecutionProvider': {'tunable_op_max_tuning_duration_ms': '0', 'enable_skip_layer_norm_strict_mode': '0', 'tunable_op_tuning_enable': '0', 'device_id': '0', 'has_user_compute_stream': '0', 'gpu_mem_limit': '18446744073709551615', 'gpu_external_alloc': '0', 'gpu_external_free': '0', 'gpu_external_empty_cache': '0', 'cudnn_conv_algo_search': 'EXHAUSTIVE', 'cudnn_conv1d_pad_to_nc1d': '0', 'arena_extend_strategy': 'kNextPowerOfTwo', 'do_copy_in_default_stream': '1', 'enable_cuda_graph': '0', 'cudnn_conv_use_max_workspace': '1', 'tunable_op_enable': '0'}}
find model: /root/autodl-tmp/ComfyUI/models/insightface/models/antelopev2/1k3d68.onnx landmark_3d_68 ['None', 3, 192, 192] 0.0 1.0
Applied providers: ['CUDAExecutionProvider', 'CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}, 'CUDAExecutionProvider': {'tunable_op_max_tuning_duration_ms': '0', 'enable_skip_layer_norm_strict_mode': '0', 'tunable_op_tuning_enable': '0', 'device_id': '0', 'has_user_compute_stream': '0', 'gpu_mem_limit': '18446744073709551615', 'gpu_external_alloc': '0', 'gpu_external_free': '0', 'gpu_external_empty_cache': '0', 'cudnn_conv_algo_search': 'EXHAUSTIVE', 'cudnn_conv1d_pad_to_nc1d': '0', 'arena_extend_strategy': 'kNextPowerOfTwo', 'do_copy_in_default_stream': '1', 'enable_cuda_graph': '0', 'cudnn_conv_use_max_workspace': '1', 'tunable_op_enable': '0'}}
find model: /root/autodl-tmp/ComfyUI/models/insightface/models/antelopev2/2d106det.onnx landmark_2d_106 ['None', 3, 192, 192] 0.0 1.0
Applied providers: ['CUDAExecutionProvider', 'CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}, 'CUDAExecutionProvider': {'tunable_op_max_tuning_duration_ms': '0', 'enable_skip_layer_norm_strict_mode': '0', 'tunable_op_tuning_enable': '0', 'device_id': '0', 'has_user_compute_stream': '0', 'gpu_mem_limit': '18446744073709551615', 'gpu_external_alloc': '0', 'gpu_external_free': '0', 'gpu_external_empty_cache': '0', 'cudnn_conv_algo_search': 'EXHAUSTIVE', 'cudnn_conv1d_pad_to_nc1d': '0', 'arena_extend_strategy': 'kNextPowerOfTwo', 'do_copy_in_default_stream': '1', 'enable_cuda_graph': '0', 'cudnn_conv_use_max_workspace': '1', 'tunable_op_enable': '0'}}
find model: /root/autodl-tmp/ComfyUI/models/insightface/models/antelopev2/genderage.onnx genderage ['None', 3, 96, 96] 0.0 1.0
Applied providers: ['CUDAExecutionProvider', 'CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}, 'CUDAExecutionProvider': {'tunable_op_max_tuning_duration_ms': '0', 'enable_skip_layer_norm_strict_mode': '0', 'tunable_op_tuning_enable': '0', 'device_id': '0', 'has_user_compute_stream': '0', 'gpu_mem_limit': '18446744073709551615', 'gpu_external_alloc': '0', 'gpu_external_free': '0', 'gpu_external_empty_cache': '0', 'cudnn_conv_algo_search': 'EXHAUSTIVE', 'cudnn_conv1d_pad_to_nc1d': '0', 'arena_extend_strategy': 'kNextPowerOfTwo', 'do_copy_in_default_stream': '1', 'enable_cuda_graph': '0', 'cudnn_conv_use_max_workspace': '1', 'tunable_op_enable': '0'}}
find model: /root/autodl-tmp/ComfyUI/models/insightface/models/antelopev2/glintr100.onnx recognition ['None', 3, 112, 112] 127.5 127.5
Applied providers: ['CUDAExecutionProvider', 'CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}, 'CUDAExecutionProvider': {'tunable_op_max_tuning_duration_ms': '0', 'enable_skip_layer_norm_strict_mode': '0', 'tunable_op_tuning_enable': '0', 'device_id': '0', 'has_user_compute_stream': '0', 'gpu_mem_limit': '18446744073709551615', 'gpu_external_alloc': '0', 'gpu_external_free': '0', 'gpu_external_empty_cache': '0', 'cudnn_conv_algo_search': 'EXHAUSTIVE', 'cudnn_conv1d_pad_to_nc1d': '0', 'arena_extend_strategy': 'kNextPowerOfTwo', 'do_copy_in_default_stream': '1', 'enable_cuda_graph': '0', 'cudnn_conv_use_max_workspace': '1', 'tunable_op_enable': '0'}}
find model: /root/autodl-tmp/ComfyUI/models/insightface/models/antelopev2/scrfd_10g_bnkps.onnx detection [1, 3, '?', '?'] 127.5 128.0
set det-size: (640, 640)
# 😺dzNodes: LayerStyle -> PersonMaskUltra V2 Processed 1 image(s).
# 😺dzNodes: LayerStyle -> PersonMaskUltra V2 Processed 1 image(s).
/root/miniconda3/lib/python3.10/site-packages/insightface/utils/transform.py:68: FutureWarning: `rcond` parameter will change to the default of machine precision times ``max(M, N)`` where M and N are the input matrix dimensions.
To use the future default and silence this warning we advise to pass `rcond=None`, to keep using the old, explicitly pass `rcond=-1`.
P = np.linalg.lstsq(X_homo, Y)[0].T # Affine matrix. 3 x 4
INFO: InsightFace detection resolution lowered to (576, 576).
INFO: InsightFace detection resolution lowered to (576, 576).
# 😺dzNodes: LayerStyle -> Processed 1 mask(s).
Requested to load ControlNet
Requested to load SDXL
Loading 2 new models
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:06<00:00, 1.99it/s]
Prompt executed in 94.32 seconds

4th back to the first work flow
got prompt
[rgthree] Using rgthree's optimized recursive execution.
model_type EPS
Using xformers attention in VAE
Using xformers attention in VAE
Requested to load SD1ClipModel
Loading 1 new model
model_path is /root/autodl-tmp/ComfyUI/custom_nodes/comfyui_controlnet_aux/ckpts/lllyasviel/Annotators/ControlNetHED.pth
Requested to load AutoencoderKL
Loading 1 new model
Requested to load BaseModel
Requested to load ControlNet
Loading 2 new models
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:04<00:00, 4.60it/s]
Prompt executed in 85.59 seconds
```

### Other

sys info

## ComfyUI-Manager: installing dependencies done.
** ComfyUI startup time: 2024-07-25 16:29:29.091806
** Platform: Linux
** Python version: 3.10.8 (main, Nov 24 2022, 14:13:03) [GCC 11.2.0]
** Python executable: /root/miniconda3/bin/python
** ComfyUI Path: /root/autodl-tmp/ComfyUI
** Log path: /root/autodl-tmp/ComfyUI/comfyui.log

Prestartup times for custom nodes:
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/rgthree-comfy
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI-Easy-Use
1.8 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI-Manager

Total VRAM 24260 MB, total RAM 386614 MB
pytorch version: 2.1.0+cu121
xformers version: 0.0.22.post4
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce RTX 3090 : cudaMallocAsync
Using xformers cross attention

[Crystools INFO] Crystools version: 1.15.0
[Crystools INFO] CPU: Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz - Arch: x86_64 - OS: Linux 5.4.0-42-generic
[Crystools INFO] Pynvml (Nvidia) initialized.
[Crystools INFO] GPU/s:
[Crystools INFO] 0) NVIDIA GeForce RTX 3090
[Crystools INFO] NVIDIA Driver: 525.105.17

[rgthree] Loaded 40 magnificent nodes.
[rgthree] Will use rgthree's optimized recursive execution.
Import times for custom nodes:
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/websocket_image_save.py
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/AIGODLIKE-COMFYUI-TRANSLATION
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/ComfyUi_NNLatentUpscale
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI_ADV_CLIP_emb
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI-WD14-Tagger
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/ComfyUi_PromptStylers
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI-AutomaticCFG
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/comfyui_controlnet_aux
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/PowerNoiseSuite
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/comfyui-portrait-master
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI-Custom-Scripts
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/ComfyMath
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/comfyui-animatediff
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/comfyui-various
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI_essentials
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/Derfuu_ComfyUI_ModdedNodes
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI-Frame-Interpolation
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI-Coziness
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/comfyui-browser
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI_IPAdapter_plus
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/rgthree-comfy
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI-KJNodes
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/efficiency-nodes-comfyui
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI_tinyterraNodes
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI-Easy-Use
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI_InstantID
0.0 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/comfyui_segment_anything
0.1 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI-Manager
0.1 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/comfyui-prompt-control
0.1 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI_smZNodes
0.1 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI-VideoHelperSuite
0.1 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI-Inspire-Pack
0.1 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI-layerdiffuse
0.5 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI_FizzNodes
0.9 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI_LayerStyle
1.8 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/was-node-suite-comfyui
2.5 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI-Crystools
3.2 seconds: /root/autodl-tmp/ComfyUI/custom_nodes/ComfyUI-Impact-Pack