Current memory manager is a garbage

### Custom Node Testing

- [x] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)

### Expected Behavior

My VRAM does not hit 12/12GB usage for SDXL model generation...

### Actual Behavior

On a simple setup of merging 2 SDXL models and then using the result for generation I'm hitting VRAM limit and app freezes after a few generations (not sure why it's so random).

It is expected only Merged model would be used for generation, only 1 CLIP as well, only 1 VAE loaded in VRAM, but it looks like EVERYTHING POSSIBLE is trying to live in VRAM. This is what you call "memory management"? Just evict everything useless for current generation run into RAM. It would take minimum time to bring it back if needed.

### Steps to Reproduce

It's specific to VRAM volume, but if you have 12GB VRAM then scenario is this: 2 checkpoints loading, CLIP and model merged, then extra LoRA (300MB in my case) after it, then default KSampler, VAE decode, save image. Simple as this, but it would take up to 12GB VRAM on VAE step (1 image 1024x1024). And even when ComfyUI can clearly see it's approaching top it does nothing even if there are so much things that can be evicted into RAM...

### Debug Logs

```powershell
Basic info about environment from log (other things feel useless for this topic):
[INFO] setup plugin alembic.autogenerate.schemas
[INFO] setup plugin alembic.autogenerate.tables
[INFO] setup plugin alembic.autogenerate.types
[INFO] setup plugin alembic.autogenerate.constraints
[INFO] setup plugin alembic.autogenerate.defaults
[INFO] setup plugin alembic.autogenerate.comments
[WARNING] Could not autodetect AIMDO implementation, assuming Nvidia
[START] Security scan
[INFO] [ComfyUI-Manager] Using uv as Python module for pip operations.
Using Python 3.14.6 environment at: venv
[DONE] Security scan
** ComfyUI startup time: 2026-08-09 09:56:20.941
** Platform: Linux
** Python version: 3.14.6 (main, Jun 23 2026, 18:14:05) [GCC 16.1.1 20260430]
** Python executable: /home/okolenmi/comfy/venv/bin/python
** ComfyUI Path: /home/okolenmi/comfy/ComfyUI
** ComfyUI Base Folder Path: /home/okolenmi/comfy/ComfyUI
** User directory: /home/okolenmi/comfy/ComfyUI/user
** ComfyUI-Manager config path: /home/okolenmi/comfy/ComfyUI/user/__manager/config.ini
** Log path: /home/okolenmi/comfy/ComfyUI/user/comfyui.log
Using Python 3.14.6 environment at: venv
Using Python 3.14.6 environment at: venv
[INFO] [PRE] ComfyUI-Manager
[INFO]
Prestartup times for custom nodes:
[INFO] 0.0 seconds: /home/okolenmi/comfy/ComfyUI/custom_nodes/rgthree-comfy
[INFO]
[INFO] Found comfy_kitchen backend eager: {'available': True, 'disabled': False, 'unavailable_reason': None, 'capabilities': ['adaln', 'apply_rope', 'apply_rope1', 'apply_rope1_', 'apply_rope_', 'apply_rope_split_half', 'apply_rope_split_half1', 'apply_rope_split_half1_', 'apply_rope_split_half_', 'convrot_w4a4_linear', 'dequantize_convrot_w4a4_weight', 'dequantize_int8_convrot_weight', 'dequantize_int8_convrot_weight_dtype', 'dequantize_int8_embedding', 'dequantize_int8_simple', 'dequantize_int8_simple_dtype', 'dequantize_mxfp8', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'gemv_awq_w4a16', 'int8_linear', 'prepare_int4_weight_for_int8_linear', 'quantize_and_rotate_rowwise', 'quantize_convrot_w4a4_weight', 'quantize_int8_convrot_weight', 'quantize_int8_rowwise', 'quantize_int8_tensorwise', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'quantize_svdquant_w4a4', 'rms_adaln', 'rms_rope', 'rms_rope1', 'rms_rope1_', 'rms_rope_', 'rms_rope_split_half', 'rms_rope_split_half1', 'rms_rope_split_half1_', 'rms_rope_split_half_', 'scaled_mm_mxfp8', 'scaled_mm_nvfp4', 'scaled_mm_svdquant_w4a4', 'stochastic_rounding_fp8']}
[INFO] Found comfy_kitchen backend cuda: {'available': False, 'disabled': True, 'unavailable_reason': 'CUDA not available on this system', 'capabilities': []}
[INFO] Found comfy_kitchen backend triton: {'available': True, 'disabled': True, 'unavailable_reason': None, 'capabilities': ['adaln', 'apply_rope', 'apply_rope1', 'apply_rope1_', 'apply_rope_', 'apply_rope_split_half', 'apply_rope_split_half1', 'apply_rope_split_half1_', 'apply_rope_split_half_', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'int8_linear', 'quantize_and_rotate_rowwise', 'quantize_int8_rowwise', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'rms_adaln', 'rms_rope', 'rms_rope1', 'rms_rope1_', 'rms_rope_', 'rms_rope_split_half', 'rms_rope_split_half1', 'rms_rope_split_half1_', 'rms_rope_split_half_']}
[INFO] Found comfy_kitchen backend hip: {'available': False, 'disabled': False, 'unavailable_reason': 'PyTorch ROCm/HIP runtime not available', 'capabilities': []}
[INFO] Checkpoint files will always be loaded safely.
[INFO] Total VRAM 12216 MB, total RAM 40046 MB
[INFO] pytorch version: 2.12.1+xpu
[INFO] Set vram state to: NORMAL_VRAM
[INFO] Device: xpu:0 Intel(R) Arc(TM) B580 Graphics
[INFO] Using pytorch attention
[INFO] Python version: 3.14.6 (main, Jun 23 2026, 18:14:05) [GCC 16.1.1 20260430]
[INFO] ComfyUI version: 0.30.0
[INFO] comfy-aimdo version: 0.4.13
[INFO] comfy-kitchen version: 0.2.26
[INFO] comfyui-frontend-package version: 1.48.6
[INFO] comfyui-workflow-templates version: 0.11.31
[INFO] comfyui-embedded-docs version: 0.5.9
[INFO] comfy-kitchen version: 0.2.26
[INFO] comfy-aimdo version: 0.4.13
```

### Other

That "freeze" after reaching VRAM limit is a specific thing for my Intel Arc B580 GPU. It looks like Nvidia have a way to safely stay near the top. CLI arguments about VRAM headroom doesn't work as well - tested them. --lowvram also doesn't help.

No source file or test is named. Start by reproducing the two-SDXL merge, LoRA, KSampler, VAE decode, and image-save workflow on an Intel Arc B580, then inspect the startup log and compare normal VRAM behavior with --lowvram and VRAM-headroom options. Done means preventing the reported VRAM-limit freeze while preserving generation.

Written by the indexing model from the issue text.