CUDA OOM Error when plenty of free memory is available

### Custom Node Testing

- [x] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)

### Expected Behavior

Generate a simple image of kittens.

### Actual Behavior

06:10:10.590 [Warning] [ComfyUI-0/STDERR] torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 MiB. GPU 0 has a total capacity of 5.79 GiB of which 3.73 GiB is free. Including non-PyTorch memory, this process has 427.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

### Steps to Reproduce

This is a fresh install of SwarmUI on a VM running a VGPU with the L40S-6Q profile. This process is normally ran through on these systems multiple times a day and just this week it has stopped working. Running flux-mini model from HuggingFace - https://huggingface.co/TencentARC/flux-mini/resolve/main/flux-mini.safetensors

### Debug Logs

```powershell
06:09:23.638 [Init] === SwarmUI v0.9.8.0 Starting at 2026-03-06 06:09:23 ===
06:09:23.640 [Init] Parsing command line...
06:09:23.653 [Init] Loading settings file...
06:09:23.691 [Init] Re-saving settings file...
06:09:23.727 [Init] Applying command line settings...
06:09:23.745 [Init] SwarmUI was installed 2026-03-06 (0 days ago) with version 0.9.8.0
06:09:23.745 [Init] Swarm base path is: /home/User/SwarmUI
06:09:23.746 [Init] Running on OS: Ubuntu 24.04.1 LTS
06:09:23.781 [Init] Prepping extension: SwarmUI.Builtin_ImageBatchToolExtension.ImageBatchToolExtension...
06:09:23.782 [Init] Prepping extension: SwarmUI.Builtin_GridGeneratorExtension.GridGeneratorExtension...
06:09:23.782 [Init] Prepping extension: SwarmUI.Builtin_DynamicThresholding.DynamicThresholdingExtension...
06:09:23.783 [Init] Prepping extension: SwarmUI.Builtin_ComfyUIBackend.ComfyUIBackendExtension...
06:09:23.783 [Init] Prepping extension: SwarmUI.Builtin_AutoWebUIExtension.AutoWebUIBackendExtension...
06:09:23.824 [Init] CPU Cores: 8 | RAM: 47.05 GiB total, 43.61 GiB available, 8.00 GiB virtual, unknown swap
06:09:23.828 [Init] Current git commit is [658ef948: other param patched], marked as date 2026-03-06 06:06:37 (6 hours ago)
06:09:23.846 [Init] Prepping options...
06:09:23.886 [Init] GPU 0: NVIDIA L40S-6Q | Temp 0C | Util 13% GPU, 2% Memory | VRAM 6.00 GiB total, 4.16 GiB free, 1.00 GiB used
06:09:23.903 [Init] Will backup user database to '/home/User/SwarmUI/Data/UsersBackups/UsersBackup_2026_9.ldb'
06:09:24.046 [Init] Swarm is up to date! You have version 0.9.8.0, and 0.9.8-Beta is the latest.
06:09:24.060 [Init] Loading models list...
06:09:24.121 [Init] Loading backends...
06:09:24.123 [Init] Loading backends from file...
06:09:24.128 [Init] Initializing backend #0 - ComfyUI Self-Starting...
06:09:24.128 [Init] Prepping API...
06:09:24.132 [Init] Prepping webserver...
06:09:24.135 [Init] Backend request handler loop ready...
06:09:24.276 [Init] Scan for web extensions...
06:09:24.277 [Init] Readying extensions for launch...
06:09:24.278 [Init] Launching server...
06:09:24.279 [Init] Starting webserver on http://localhost:7801
06:09:24.335 [Init] SwarmUI v0.9.8.0 - Local is now running.
06:09:24.350 [Init] Self-Start ComfyUI-0 on port 7821 is loading...
06:09:24.836 [Init] Launch web browser...
Gtk-Message: 06:09:24.987: Not loading module "atk-bridge": The functionality is provided by GTK natively. Please try to not load it.
[27068, Main Thread] WARNING: GTK+ module /snap/firefox/7901/gnome-platform/usr/lib/gtk-2.0/modules/libcanberra-gtk-module.so cannot be loaded.
GTK+ 2.x symbols detected. Using GTK+ 2.x and GTK+ 3 in the same process is not supported.: 'glib warning', file /build/firefox/parts/firefox/build/toolkit/xre/nsSigHandlers.cpp:201

(firefox_firefox:27068): Gtk-WARNING **: 06:09:25.043: GTK+ module /snap/firefox/7901/gnome-platform/usr/lib/gtk-2.0/modules/libcanberra-gtk-module.so cannot be loaded.
GTK+ 2.x symbols detected. Using GTK+ 2.x and GTK+ 3 in the same process is not supported.
Gtk-Message: 06:09:25.043: Failed to load module "canberra-gtk-module"
[27068, Main Thread] WARNING: GTK+ module /snap/firefox/7901/gnome-platform/usr/lib/gtk-2.0/modules/libcanberra-gtk-module.so cannot be loaded.
GTK+ 2.x symbols detected. Using GTK+ 2.x and GTK+ 3 in the same process is not supported.: 'glib warning', file /build/firefox/parts/firefox/build/toolkit/xre/nsSigHandlers.cpp:201

(firefox_firefox:27068): Gtk-WARNING **: 06:09:25.044: GTK+ module /snap/firefox/7901/gnome-platform/usr/lib/gtk-2.0/modules/libcanberra-gtk-module.so cannot be loaded.
GTK+ 2.x symbols detected. Using GTK+ 2.x and GTK+ 3 in the same process is not supported.
Gtk-Message: 06:09:25.044: Failed to load module "canberra-gtk-module"
06:09:25.511 [Info] Creating new session 'local' for 127.0.0.1
06:09:30.730 [Init] Self-Start ComfyUI-0 on port 7821 started.
06:09:49.104 [Info] User local requested 1 image with model 'flux-mini.safetensors'...
06:10:10.586 [Warning] [ComfyUI-0/STDERR] Traceback (most recent call last):
06:10:10.587 [Warning] [ComfyUI-0/STDERR] File "/home/User/SwarmUI/dlbackend/ComfyUI/execution.py", line 524, in execute
06:10:10.587 [Warning] [ComfyUI-0/STDERR] output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
06:10:10.587 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
06:10:10.587 [Warning] [ComfyUI-0/STDERR] File "/home/User/SwarmUI/dlbackend/ComfyUI/execution.py", line 333, in get_output_data
06:10:10.587 [Warning] [ComfyUI-0/STDERR] return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
06:10:10.587 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
06:10:10.587 [Warning] [ComfyUI-0/STDERR] File "/home/User/SwarmUI/dlbackend/ComfyUI/execution.py", line 307, in _async_map_node_over_list
06:10:10.588 [Warning] [ComfyUI-0/STDERR] await process_inputs(input_dict, i)
06:10:10.588 [Warning] [ComfyUI-0/STDERR] File "/home/User/SwarmUI/dlbackend/ComfyUI/execution.py", line 295, in process_inputs
06:10:10.588 [Warning] [ComfyUI-0/STDERR] result = f(**inputs)
06:10:10.588 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^
06:10:10.588 [Warning] [ComfyUI-0/STDERR] File "/home/User/SwarmUI/src/BuiltinExtensions/ComfyUIBackend/ExtraNodes/SwarmComfyCommon/SwarmKSampler.py", line 382, in run_sampling
06:10:10.588 [Warning] [ComfyUI-0/STDERR] return self.sample(model, noise_seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, start_at_step, end_at_step, var_seed, var_seed_strength, sigma_max, sigma_min, rho, add_noise, return_with_leftover_noise, previews)
06:10:10.589 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
06:10:10.589 [Warning] [ComfyUI-0/STDERR] File "/home/User/SwarmUI/src/BuiltinExtensions/ComfyUIBackend/ExtraNodes/SwarmComfyCommon/SwarmKSampler.py", line 356, in sample
06:10:10.589 [Warning] [ComfyUI-0/STDERR] samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_samples,
06:10:10.589 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
06:10:10.589 [Warning] [ComfyUI-0/STDERR] File "/home/User/SwarmUI/dlbackend/ComfyUI/comfy/sample.py", line 64, in sample
06:10:10.589 [Warning] [ComfyUI-0/STDERR] sampler = comfy.samplers.KSampler(model, steps=steps, device=model.load_device, sampler=sampler_name, scheduler=scheduler, denoise=denoise, model_options=model.model_options)
06:10:10.589 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
06:10:10.589 [Warning] [ComfyUI-0/STDERR] File "/home/User/SwarmUI/dlbackend/ComfyUI/comfy/samplers.py", line 1129, in __init__
06:10:10.590 [Warning] [ComfyUI-0/STDERR] self.set_steps(steps, denoise)
06:10:10.590 [Warning] [ComfyUI-0/STDERR] File "/home/User/SwarmUI/dlbackend/ComfyUI/comfy/samplers.py", line 1150, in set_steps
06:10:10.590 [Warning] [ComfyUI-0/STDERR] self.sigmas = self.calculate_sigmas(steps).to(self.device)
06:10:10.590 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
06:10:10.590 [Warning] [ComfyUI-0/STDERR] torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 MiB. GPU 0 has a total capacity of 5.79 GiB of which 3.73 GiB is free. Including non-PyTorch memory, this process has 427.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
06:10:10.590 [Warning] [ComfyUI-0/STDERR]
```

### Other

I normally only run the following backend... --disable-cuda-malloc

But for the purpose of troubleshooting I've also added --lowvram --force-channels-last --use-pytorch-cross-attention

Not previously the error was for ~125MiB.. since adding the additional ExtraArgs it's only trying to pull 2.00MiB and still failing.