VRAM allocation

### Expected Behavior

Python version: 3.12.9 (tags/v3.12.9:fdb8142, Feb 4 2025, 15:27:58) [MSC v.1942 64 bit (AMD64)]
ComfyUI version: 0.3.33
ComfyUI frontend version: 1.18.9
RTX 4090
driver versions 576.02 and 576.28 (reinstalled older version to see if nvidia broke something, nope)
comfy and comfy fp16 version

It seems something has broken vram allocation, flux models (dev fp16,redux which uses devfp16) (https://comfyanonymous.github.io/ComfyUI_examples/flux/) width 1088, height 1344, batch 4. now start using shared GPU ram in the GB's which they didn't before. This causes a massive slow down in it/s (now 75s/it before 8s/it) and effectively make them unusable. I have ran a modified version of the above fp16 workflow with ~50 loras (power lora loader) loaded and it wasn't this slow.

Is this a python or comfy issue?

### Actual Behavior

massive increase in s/it

### Steps to Reproduce

have a 4090, (or a 3090 i guess)
run the above fp16 workflow with image size and batch stated,
watch shared GPU memory in task manager

### Debug Logs

```powershell
K:\ComfyUI>.\python_embeded\python.exe -s ComfyUI\main.py --windows-standalone-build
Total VRAM 24564 MB, total RAM 32694 MB
pytorch version: 2.7.0+cu128
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce RTX 4090 : native
Checkpoint files will always be loaded safely.
Using pytorch attention
[START] Security scan
[DONE] Security scan
## ComfyUI-Manager: installing dependencies done.
** ComfyUI startup time: 2025-05-08 18:51:05.085
** Platform: Windows
** Python version: 3.12.9 (tags/v3.12.9:fdb8142, Feb 4 2025, 15:27:58) [MSC v.1942 64 bit (AMD64)]
** Python executable: K:\ComfyUI\python_embeded\python.exe
** ComfyUI Path: K:\ComfyUI\ComfyUI
** ComfyUI Base Folder Path: K:\ComfyUI\ComfyUI
** User directory: K:\ComfyUI\ComfyUI\user
** ComfyUI-Manager config path: K:\ComfyUI\ComfyUI\user\default\ComfyUI-Manager\config.ini
** Log path: K:\ComfyUI\ComfyUI\user\comfyui.log

Prestartup times for custom nodes:
0.0 seconds: K:\ComfyUI\ComfyUI\custom_nodes\rgthree-comfy
0.0 seconds: K:\ComfyUI\ComfyUI\custom_nodes\comfyui-easy-use
2.4 seconds: K:\ComfyUI\ComfyUI\custom_nodes\comfyui-manager
8.5 seconds: K:\ComfyUI\ComfyUI\custom_nodes\agilly1989_motorway

Python version: 3.12.9 (tags/v3.12.9:fdb8142, Feb 4 2025, 15:27:58) [MSC v.1942 64 bit (AMD64)]
ComfyUI version: 0.3.33
ComfyUI frontend version: 1.18.9
[Prompt Server] web root: K:\ComfyUI\python_embeded\Lib\site-packages\comfyui_frontend_package\static
------------------------------------------
Comfyroll Studio v1.76 : 175 Nodes Loaded
------------------------------------------
** For changes, please see patch notes at https://github.com/Suzie1/ComfyUI_Comfyroll_CustomNodes/blob/main/Patch_Notes.md
** For help, please see the wiki at https://github.com/Suzie1/ComfyUI_Comfyroll_CustomNodes/wiki
------------------------------------------
[ComfyUI-Easy-Use] server: v1.2.9 Loaded
[ComfyUI-Easy-Use] web root: K:\ComfyUI\ComfyUI\custom_nodes\comfyui-easy-use\web_version/v1 Loaded
### Loading: ComfyUI-Impact-Pack (V8.14.2)
[Impact Pack] Wildcards loading done.
K:\ComfyUI\python_embeded\Lib\site-packages\albumentations\__init__.py:13: UserWarning: A new version of Albumentations is available: 2.0.6 (you have 1.4.15). Upgrade using: pip install -U albumentations. To disable automatic update checks, set the environment variable NO_ALBUMENTATIONS_UPDATE to 1.
check_for_updates()
K:\ComfyUI\python_embeded\Lib\site-packages\timm\models\layers\__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
### Loading: ComfyUI-Manager (V3.31.13)
[ComfyUI-Manager] network_mode: public
### ComfyUI Version: v0.3.33-1-g924d771e | Released on '2025-05-08'
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/alter-list.json
2025-05-08 18:51:10.940 | DEBUG | jovi_glsl.core::40 - user shader folder: K:\ComfyUI\ComfyUI\custom_nodes\jovi_glsl\user
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/model-list.json
2025-05-08 18:51:10.956 | INFO | jovi_glsl.core::71 - vertex programs: 0
2025-05-08 18:51:10.956 | INFO | jovi_glsl.core::71 - vertex programs: 0
2025-05-08 18:51:10.959 | INFO | jovi_glsl.core::72 - fragment programs: 45
2025-05-08 18:51:10.959 | INFO | jovi_glsl.core::72 - fragment programs: 45
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/github-stats.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/extension-node-map.json
No OpenGL_accelerate module loaded: No module named 'OpenGL_accelerate'
2025-05-08 18:51:11.533 | INFO | cozy_comfyui.node:loader:121 - JOV_GL 45 nodes loaded
2025-05-08 18:51:11.533 | INFO | cozy_comfyui.node:loader:121 - JOV_GL 45 nodes loaded
2025-05-08 18:51:11.553 | INFO | cozy_comfyui.node:loader:121 - JOV_GL 45 nodes loaded
2025-05-08 18:51:11.553 | INFO | cozy_comfyui.node:loader:121 - JOV_GL 45 nodes loaded

[rgthree-comfy] Loaded 42 extraordinary nodes. 🎉

WAS Node Suite: OpenCV Python FFMPEG support is enabled
WAS Node Suite Warning: `ffmpeg_bin_path` is not set in `K:\ComfyUI\ComfyUI\custom_nodes\was-node-suite-comfyui\was_suite_config.json` config file. Will attempt to use system ffmpeg binaries if available.
WAS Node Suite: Finished. Loaded 220 nodes successfully.

"The only way to do great work is to love what you do." - Steve Jobs

Import times for custom nodes:
0.0 seconds: K:\ComfyUI\ComfyUI\custom_nodes\websocket_image_save.py
0.0 seconds: K:\ComfyUI\ComfyUI\custom_nodes\comfyui-img2paintingassistant
0.0 seconds: K:\ComfyUI\ComfyUI\custom_nodes\stability-ComfyUI-nodes
0.0 seconds: K:\ComfyUI\ComfyUI\custom_nodes\comfyui-custom-scripts
0.0 seconds: K:\ComfyUI\ComfyUI\custom_nodes\comfyui_essentials
0.0 seconds: K:\ComfyUI\ComfyUI\custom_nodes\rgthree-comfy
0.1 seconds: K:\ComfyUI\ComfyUI\custom_nodes\ComfyUI_Comfyroll_CustomNodes
0.1 seconds: K:\ComfyUI\ComfyUI\custom_nodes\comfyui-impact-pack
0.1 seconds: K:\ComfyUI\ComfyUI\custom_nodes\agilly1989_motorway
0.2 seconds: K:\ComfyUI\ComfyUI\custom_nodes\comfyui-easy-use
0.2 seconds: K:\ComfyUI\ComfyUI\custom_nodes\comfyui-manager
0.5 seconds: K:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-YOLO
0.7 seconds: K:\ComfyUI\ComfyUI\custom_nodes\jovi_glsl
1.1 seconds: K:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-Lotus
1.3 seconds: K:\ComfyUI\ComfyUI\custom_nodes\comfyui-inspyrenet-rembg
1.4 seconds: K:\ComfyUI\ComfyUI\custom_nodes\was-node-suite-comfyui

Starting server

To see the GUI go to: http://127.0.0.1:8188
FETCH ComfyRegistry Data: 5/84
FETCH ComfyRegistry Data: 10/84
FETCH ComfyRegistry Data: 15/84
FETCH ComfyRegistry Data: 20/84
FETCH ComfyRegistry Data: 25/84
FETCH ComfyRegistry Data: 30/84
FETCH ComfyRegistry Data: 35/84
FETCH ComfyRegistry Data: 40/84
FETCH ComfyRegistry Data: 45/84
FETCH ComfyRegistry Data: 50/84
FETCH ComfyRegistry Data: 55/84
FETCH ComfyRegistry Data: 60/84
FETCH ComfyRegistry Data: 65/84
FETCH ComfyRegistry Data: 70/84
FETCH ComfyRegistry Data: 75/84
FETCH ComfyRegistry Data: 80/84
FETCH ComfyRegistry Data [DONE]
[ComfyUI-Manager] default cache updated: https://api.comfy.org/nodes
FETCH DATA from: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json [DONE]
[ComfyUI-Manager] All startup tasks have been completed.
got prompt
Using pytorch attention in VAE
Using pytorch attention in VAE
VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
model weight dtype torch.bfloat16, manual cast: None
model_type FLUX
Requested to load FluxClipModel_
loaded completely 9.5367431640625e+25 9319.23095703125 True
CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cuda:0, dtype: torch.float16
clip missing: ['text_projection.weight']
Requested to load Flux
loaded partially 17195.923 17195.73828125 0
```

### Other

_No response_