With SD (3.5 and XL) and MacOS, performance got much worse from latest release v0.2.6 to current master

### Expected Behavior

When I pull master, I get the same or better performance as with the revision that I had before.

### Actual Behavior

I used to be able to run SD3.5 at around 7 seconds per iteration, but when I pulled master which is currently on revision fabf449febfc71f863f48052d83bfed7b1727a45 (Mochi VAE encoder.), the performance got worse, up to 11.5 seconds per iteration. I rolled back ComfyUI to the latest release version (0.2.6), and the performance was back to 7s per iteration.

### Steps to Reproduce

Check out version v2.6.0, and run SD3.5.
Check out revision fabf449febfc71f863f48052d83bfed7b1727a45, and run SD3.5.
Performance should be considerably different.

### Debug Logs

```powershell
Running MacOS 15 on Apple silicon
Pytorch 2.6.0.dev-20241101
Workflow is the first example from https://comfyanonymous.github.io/ComfyUI_examples/sd3/, but with t5xxl_fp8_e4m3fn.safetensors and sd3.5_medium.safetensors
```

### Other

_No response_