Since making LoRA work on NVFP4 models, VRAM exhaustion on 40-series arises when using a mixed-precision model with LoRA

### Custom Node Testing

- [x] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)

### Expected Behavior

Since this commit https://github.com/Comfy-Org/ComfyUI/commit/b3c0e4de57bfd27e3dd94bd9723bb4c714668a09,
Regardless of the resolution, when using the qwen2511 fp8mixed model with LORAs or the ltx2 fp8 model with LORAs, the `Allocation on device` error will occur.
However, models with other precision normally, and the 5090 also operates without issues. This problem occurs exclusively with the 4090

### Actual Behavior

https://github.com/user-attachments/assets/0373b7bb-2fb4-46b1-8ea1-c53f6cae56ae

### Steps to Reproduce

1. Running the latest version of ComfyUI on the 4090
2. Run this workflow [qwen edit 2511 fp8mixed test.json](https://github.com/user-attachments/files/24757273/qwen.edit.2511.fp8mixed.test.json) or ltx2's workflow.

### Debug Logs

```powershell
F:\ComfyUI-aki-v2\python\Lib\site-packages\torch\cuda\__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
Checkpoint files will always be loaded safely.
Total VRAM 49140 MB, total RAM 95800 MB
pytorch version: 2.9.1+cu130
xformers version: 0.0.33.post2
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce RTX 4090 : cudaMallocAsync
Using async weight offloading with 2 streams
Enabled pinned memory 43110.0
working around nvidia conv3d memory bug.
Found comfy_kitchen backend triton: {'available': True, 'disabled': True, 'unavailable_reason': None, 'capabilities': ['apply_rope', 'apply_rope1', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8']}
Found comfy_kitchen backend cuda: {'available': True, 'disabled': False, 'unavailable_reason': None, 'capabilities': ['apply_rope', 'apply_rope1', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'scaled_mm_nvfp4']}
Found comfy_kitchen backend eager: {'available': True, 'disabled': False, 'unavailable_reason': None, 'capabilities': ['apply_rope', 'apply_rope1', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'scaled_mm_nvfp4']}
Using xformers attention
Python version: 3.12.10 (tags/v3.12.10:0cc8128, Apr 8 2025, 12:21:36) [MSC v.1943 64 bit (AMD64)]
ComfyUI version: 0.10.0
Initializing frontend: Comfy-Org/ComfyUI_frontend@latest, requesting version details from GitHub...
[Prompt Server] web root: F:\ComfyUI-aki-v2\ComfyUI\web_custom_versions\Comfy-Org_ComfyUI_frontend\1.38.8
Skipping loading of custom nodes
Context impl SQLiteImpl.
Will assume non-transactional DDL.
Assets scan(roots=['models']) completed in 0.223s (created=0, skipped_existing=12, total_seen=613)
Starting server

To see the GUI go to: http://0.0.0.0:8188
got prompt
Using xformers attention in VAE
Using xformers attention in VAE
VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
Requested to load WanVAE
loaded completely; 44889.34 MB usable, 242.03 MB loaded, full load: True
Found quantization metadata version 1
Using MixedPrecisionOps for text encoder
Requested to load QwenImageTEModel_
loaded completely; 7910.28 MB loaded, full load: True
CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cuda:0, dtype: torch.float16
Found quantization metadata version 1
Detected mixed precision quantization
Using mixed precision operations
model weight dtype torch.bfloat16, manual cast: torch.bfloat16
model_type FLUX
Requested to load QwenImage
ERROR lora diffusion_model.transformer_blocks.1.img_mlp.net.2.weight Allocation on device
ERROR lora diffusion_model.transformer_blocks.1.img_mlp.net.2.weight Allocation on device
!!! Exception during processing !!! Allocation on device
```

### Other

_No response_