Kontext doesn't work well with lora when rank is large

### Custom Node Testing

- [x] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)

### Expected Behavior

Kontext should work well with the lora loader, whatever the lora rank is

### Actual Behavior

When the lora rank is 4 or 8, the results look great. However, when the lora rank is 16 or 32, the results become to noise. I also tried the same lora with diffusers, and everything was fine. Btw, if I used diffusers to merge lora into the transformer and then loaded this model in comfyui, the results would also be correct.

### Steps to Reproduce

None

### Debug Logs

```powershell
None
```

### Other

_No response_