RX 7900 XTX (maybe AMD in general) render time build-up issue

### Expected Behavior

I am rendering the same size etc. over and over expecting only minor changes in time related to prompts.

### Actual Behavior

As more prompts are iterated over render time accumulates.

Clearing cache helped this significantly normally it would build up tremendously over 10 queue items, but when clearing cache after every queue item it only does this over 100s only adding about 20/30% render times over 300 queue items. when not clearing cache it builds up to 2-3x longer render times then initial queue items over only 10 items.

So although improving it still accumulates time and I don't know why.

Since I render 1000s or 10's of 1000's of queue items at a time this does remain an issue since it might add hours or days to my final result (the combined time from going through 1000's of these) for no reason.

### Steps to Reproduce

Use amd

### Debug Logs

```powershell
/
```

### Other

_No response_