Higher GPU usage, with no performance gains

### Expected Behavior

Don't have extra GPU usage, if it does not improve performance.

### Actual Behavior

I have a GTX 1070.
My GPU usage was always low in the KSampler stage ( less than 5 % ).

GPU usage was also low with Flux, when it was first implemented. 
But after some updates, the GPU usage got higher ( about 60 % ).

At first thought this was related to the Flux speed improvements.
Because, currently I have faster both performance and higher GPU usage, than when Flux was first implemented.

But I remember using some past Comfy UI versions ( maybe a month ago ) which had low GPU usage ( less than 5 % ) and the same fast performance.
 
I don't know the exact versions that gave fast performance, with low GPU usage, because I use GGUF models more regularly. 
And currently, with GGUF models, I always have fast performance and low GPU usage.
I only noticed this issue recently, when I tried a Flux FP8 fine tune. 

### Steps to Reproduce

.

### Debug Logs

```powershell
.
```

### Other

Windows 10

```
Total VRAM 8192 MB, total RAM 32727 MB
pytorch version: 2.3.1+cu121
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce GTX 1070 : cudaMallocAsync
```