Flux.2 inference slower after commit 0c18842a [v0.3.73]

### Custom Node Testing

- [x] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)

### Expected Behavior

Commit 0c18842 [v0.3.73](https://github.com/comfyanonymous/ComfyUI/releases/tag/v0.3.73)
```
Checkpoint files will always be loaded safely.
Total VRAM 22104 MB, total RAM 93801 MB
pytorch version: 2.10.0a0+rocm7.11.0a20251124
Set: torch.backends.cudnn.enabled = False for better AMD performance.
AMD arch: gfx1100
ROCm version: (7, 2)
Set vram state to: NORMAL_VRAM
Disabling smart memory management
Device: cuda:0 AMD Radeon RX 7900 XTX : native
Enabled pinned memory 42210.0
Using pytorch attention
Python version: 3.12.10 (tags/v3.12.10:0cc8128, Apr 8 2025, 12:21:36) [MSC v.1943 64 bit (AMD64)]
ComfyUI version: 0.3.73
ComfyUI frontend version: 1.30.6
[Prompt Server] web root: D:\ComfyUI\venv\Lib\site-packages\comfyui_frontend_package\static
Total VRAM 22104 MB, total RAM 93801 MB
pytorch version: 2.10.0a0+rocm7.11.0a20251124
Set: torch.backends.cudnn.enabled = False for better AMD performance.
AMD arch: gfx1100
ROCm version: (7, 2)
Set vram state to: NORMAL_VRAM
Disabling smart memory management
Device: cuda:0 AMD Radeon RX 7900 XTX : native
Enabled pinned memory 42210.0
Skipping loading of custom nodes
Context impl SQLiteImpl.
Will assume non-transactional DDL.
No target revision found.
Using RAM pressure cache.
Starting server

To see the GUI go to: http://127.0.0.1:8188
got prompt
Using split attention in VAE
Using split attention in VAE
VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
Requested to load AutoencoderKL
loaded completely; 11762.26 MB usable, 160.31 MB loaded, full load: True
Using MixedPrecisionOps for text encoder: 210 quantized layers
CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
Requested to load Flux2TEModel_
loaded completely; 20291.24 MB usable, 17180.59 MB loaded, full load: True
Found quantization metadata (version 1.0)
Detected mixed precision quantization: 128 layers quantized
Using mixed precision operations: 128 quantized layers
model weight dtype torch.float8_e4m3fn, manual cast: torch.bfloat16
model_type FLUX
Requested to load Flux2
0 models unloaded.
loaded partially; 8689.77 MB usable, 8649.02 MB loaded, 25164.00 MB offloaded, lowvram patches: 0
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [04:04<00:00, 12.25s/it]
Requested to load AutoencoderKL
loaded completely; 9118.00 MB usable, 160.31 MB loaded, full load: True
Exception in callback _ProactorBasePipeTransport._call_connection_lost(None)
handle:
Traceback (most recent call last):
File "C:\Users\sarmi\AppData\Local\Programs\Python\Python312\Lib\asyncio\events.py", line 88, in _run
self._context.run(self._callback, *self._args)
File "C:\Users\sarmi\AppData\Local\Programs\Python\Python312\Lib\asyncio\proactor_events.py", line 165, in _call_connection_lost
self._sock.shutdown(socket.SHUT_RDWR)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host
Prompt executed in 302.17 seconds
```

### Actual Behavior

commit 8402c870 [v0.3.75](https://github.com/comfyanonymous/ComfyUI/releases/tag/v0.3.75)
```
Checkpoint files will always be loaded safely.
Total VRAM 22104 MB, total RAM 93801 MB
pytorch version: 2.10.0a0+rocm7.11.0a20251124
Set: torch.backends.cudnn.enabled = False for better AMD performance.
AMD arch: gfx1100
ROCm version: (7, 2)
Set vram state to: NORMAL_VRAM
Disabling smart memory management
Device: cuda:0 AMD Radeon RX 7900 XTX : native
Enabled pinned memory 42210.0
Using pytorch attention
Python version: 3.12.10 (tags/v3.12.10:0cc8128, Apr 8 2025, 12:21:36) [MSC v.1943 64 bit (AMD64)]
ComfyUI version: 0.3.75
ComfyUI frontend version: 1.30.6
[Prompt Server] web root: D:\ComfyUI\venv\Lib\site-packages\comfyui_frontend_package\static
Total VRAM 22104 MB, total RAM 93801 MB
pytorch version: 2.10.0a0+rocm7.11.0a20251124
Set: torch.backends.cudnn.enabled = False for better AMD performance.
AMD arch: gfx1100
ROCm version: (7, 2)
Set vram state to: NORMAL_VRAM
Disabling smart memory management
Device: cuda:0 AMD Radeon RX 7900 XTX : native
Enabled pinned memory 42210.0
Skipping loading of custom nodes
Context impl SQLiteImpl.
Will assume non-transactional DDL.
No target revision found.
Using RAM pressure cache.
Starting server

To see the GUI go to: http://127.0.0.1:8188
got prompt
Using split attention in VAE
Using split attention in VAE
VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
Requested to load AutoencoderKL
loaded completely; 11762.26 MB usable, 160.31 MB loaded, full load: True
Using MixedPrecisionOps for text encoder: 210 quantized layers
CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
Requested to load Flux2TEModel_
loaded completely; 20291.24 MB usable, 17180.59 MB loaded, full load: True
Found quantization metadata (version 1.0)
Detected mixed precision quantization: 128 layers quantized
Using mixed precision operations: 128 quantized layers
model weight dtype torch.float8_e4m3fn, manual cast: torch.bfloat16
model_type FLUX
Requested to load Flux2
loaded partially; 16268.45 MB usable, 16245.02 MB loaded, 17568.00 MB offloaded, lowvram patches: 0
70%|███████████████████████████████████████████████████████████████████████████████████████▌ | 14/20 [04:07<01:46, 17.68s/it]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [05:54<00:00, 17.72s/it]
Requested to load AutoencoderKL
loaded completely; 9118.00 MB usable, 160.31 MB loaded, full load: True
Prompt executed in 418.58 seconds
```

### Steps to Reproduce

[image_flux2_fp8 (1).json](https://github.com/user-attachments/files/23777543/image_flux2_fp8.1.json)

### Debug Logs

```powershell
.
```

### Other

After release 0c18842a Flux2 inference got slower with higher VRAM usage. from: ~12s/it to ~18s/it | from ~17GBVRAM to ~22GB VRAM

At least on my machine.

`python main.py --use-pytorch-cross-attention --disable-smart-memory --mmap-torch-files --disable-api-nodes --normalvram --cache-ram 1000 --disable-all-custom-nodes`