Decreased inference speed after latest changes

### Expected Behavior

The inference speed and VRAM usage should have remained the same.

### Actual Behavior

The inference speed is about 20% slower and VRAM usage is lower as well.

### Steps to Reproduce

Used a default SDXL workflow with lora

### Debug Logs

```powershell
No error logs
```

### Other

_No response_