Very inconsistent text encoding for ace step 1.5 with 4B Lm

### Custom Node Testing

- [ ] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)

### Expected Behavior

loaded partially; 5341.79 MB usable, 5104.29 MB loaded, 4023.00 MB offloaded, 237.50 MB buffer reserved, lowvram patches: 0
LM sampling: 100%|███████████████████████████████████████████████████████████████████| 975/975 [13:16<00:00, 1.22it/s]

### Actual Behavior

loaded partially; 5341.79 MB usable, 5104.29 MB loaded, 4023.00 MB offloaded, 237.50 MB buffer reserved, lowvram patches: 0
LM sampling: 46%|██████████████████████████████▊ | 448/975 [10:55<26:45, 3.05s/it]

### Steps to Reproduce

Load the text encoders using DualCLIPLoaderMultiGPU

### Debug Logs

```powershell
got prompt
Requested to load ACE15TEModel_
loaded partially; 5341.79 MB usable, 5104.29 MB loaded, 4023.00 MB offloaded, 237.50 MB buffer reserved, lowvram patches: 0
LM sampling: 100%|███████████████████████████████████████████████████████████████████| 975/975 [13:16<00:00, 1.22it/s]
model weight dtype torch.bfloat16, manual cast: torch.float32
model_type FLOW
Error collecting metadata (pre-execution): tuple index out of range
Requested to load ACEStep15
loaded completely; 5244.49 MB usable, 4565.35 MB loaded, full load: True
100%|██████████████████████████████████████████████████████████████████████████████████| 16/16 [00:52<00:00, 3.26s/it]
Requested to load AudioOobleckVAE
0 models unloaded.
loaded completely; 321.70 MB loaded, full load: True
Prompt executed in 00:16:06
got prompt
Potential memory leak detected with model ACE15TEModel_, doing a full garbage collect, for maximum performance avoid circular references in the model code.
Requested to load ACE15TEModel_
loaded partially; 5341.79 MB usable, 5104.29 MB loaded, 4023.00 MB offloaded, 237.50 MB buffer reserved, lowvram patches: 0
LM sampling: 46%|██████████████████████████████▊ | 448/975 [10:55<26:45, 3.05s/it]
```

### Other

Very inconsistent speed at of text enoding with the 4b lm.
Please take note of the units
[13:16<00:00, 1.22it/s] (When the speed unit shows it/s this is the normal behavior)

[10:55<26:45, 3.05s/it] (When the speed unit shows s/it this is when the text encoding takes awfully long time sometimes even over 1 hour )