--`reserve-vram` is reserving too much VRAM

### Expected Behavior

The amount of reserved VRAM stays constant during generations, and proportional to the amount specified in the cmd.

### Actual Behavior

With `--reserve-vram 1.2` I usually have 6.5 ~ 7 GB VRAM usage, during the KSampler stage ( I have an 8 GB card ).

Now, after some generations the VRAM usage decreased to 3.4 GB

The generation speed did not decrease significantly, but I never noticed this before.


### Steps to Reproduce

I need `--reserve-vram` to use Flux ControlNets.

I was using `--reserve-vram 1.2`, with Flux Q4_K_S, some loras, and Flux Depth ControlNets ( Shaker AI and Jasper AI ).

I only used one ControlNet at the time, but I was switching between both of them.
I also canceled some generations, just to check some settings faster, because my GPU is slow.

After some generations, I noticed I was at only 3.4 GB usage.

### Debug Logs

```powershell
.
```

### Other

Windows 10
Portable Comfy Install
```
Total VRAM 8192 MB, total RAM 32727 MB
pytorch version: 2.3.1+cu121
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce GTX 1070 : cudaMallocAsync
```