manual cast: torch.bfloat16 when using fp8 combined flux.dev models causing vram issues with LoRAs

### Expected Behavior

When using separate loaders for unet, clip and vae, in the console it says: model weight dtype torch.bfloat16, manual cast: None which is expected behavior for the combined models too. The fp8 combo models are: flux1.dev fp8, clip_l, and t5xxl fp8 e4m3fn.

### Actual Behavior

model weight dtype torch.float8_e4m3fn, manual cast: torch.bfloat16
This doesn't occur when using separate loaders, only the combined/checkpoint one. This causes significant VRAM issues when using LoRAs

### Steps to Reproduce

Download a combined flux dev model, such as: https://huggingface.co/Comfy-Org/flux1-dev
Load the model using the checkpoint loader or similar.
Observe the problems in the console.
Additionally using any Ostris Lora will give OOM errors.

### Debug Logs

```powershell
2024-08-15 01:49:49.382 [ComfyUI-0] [STDERR] To see the GUI go to: http://127.0.0.1:7823
2024-08-15 01:50:28.189 [ComfyUI-0] [STDERR] got prompt
2024-08-15 01:50:28.500 [ComfyUI-0] [STDERR] model weight dtype torch.float8_e4m3fn, manual cast: torch.bfloat16
2024-08-15 01:50:28.503 [ComfyUI-0] [STDERR] model_type FLUX
2024-08-15 01:50:47.522 [ComfyUI-0] [STDERR] Using pytorch attention in VAE
2024-08-15 01:50:47.524 [ComfyUI-0] [STDERR] Using pytorch attention in VAE
2024-08-15 01:50:48.388 [ComfyUI-0] [STDERR] Requested to load FluxClipModel_
2024-08-15 01:50:48.388 [ComfyUI-0] [STDERR] Loading 1 new model
2024-08-15 01:50:57.779 [ComfyUI-0] [STDERR] loaded straight to GPU
2024-08-15 01:50:57.779 [ComfyUI-0] [STDERR] Requested to load Flux
2024-08-15 01:50:57.779 [ComfyUI-0] [STDERR] Loading 1 new model
2024-08-15 01:51:02.870 [ComfyUI-0] [STDERR] Prompt executed in 34.68 seconds
2024-08-15 01:51:03.369 [ComfyUI-0] [STDERR] got prompt
2024-08-15 01:51:03.584 [ComfyUI-0] [STDERR] model weight dtype torch.float8_e4m3fn, manual cast: torch.bfloat16
2024-08-15 01:51:03.585 [ComfyUI-0] [STDERR] model_type FLUX
2024-08-15 01:51:24.262 [ComfyUI-0] [STDERR] Using pytorch attention in VAE
2024-08-15 01:51:24.267 [ComfyUI-0] [STDERR] Using pytorch attention in VAE
2024-08-15 01:51:25.453 [ComfyUI-0] [STDERR] Requested to load FluxClipModel_
2024-08-15 01:51:25.453 [ComfyUI-0] [STDERR] Loading 1 new model
2024-08-15 01:51:33.667 [ComfyUI-0] [STDERR] loaded straight to GPU
2024-08-15 01:51:33.667 [ComfyUI-0] [STDERR] Requested to load Flux
2024-08-15 01:51:33.667 [ComfyUI-0] [STDERR] Loading 1 new model
2024-08-15 01:51:39.680 [ComfyUI-0] [STDERR] Requested to load FluxClipModel_
2024-08-15 01:51:39.680 [ComfyUI-0] [STDERR] Loading 1 new model
```

### Other

_No response_