[Bug]: VRAM leak if LoRA is added to prompt

Nobody has claimed this yet.

### Checklist

- [ ] The issue exists after disabling all extensions
- [ ] The issue exists on a clean installation of webui
- [ ] The issue is caused by an extension, but I believe it is caused by a bug in the webui
- [X] The issue exists in the current version of the webui
- [X] The issue has not been reported before recently
- [ ] The issue has been reported before but has not been fixed yet

### What happened?

*) The problem appeared right after the update, after which the 'Never OOM Integrated' tab was added. Maybe this will help.

The problem appears always after another image generation:
If LoRA (any size) is added to the prompt before the next image generation, VRAM is leaked, all memory is filled, system shared memory is used and generation slows down ten times.

And after generation is complete, memory is not cleared and you can continue working only by performing one of these actions:
- Restart the entire WebUI
- Change the model to some other model and then back again.
- Enable the 'Enabled for UNet' checkbox on the 'Never OOM Integrated' tab so that the script clears memory beforehand during the next image generation.
- Remove LoRA from the prompt. But the problem will happen again if you put it back in.

### Steps to reproduce the problem

How to reproduce the bug:
_I'll accompany each step with screenshots of VRAM loadings._
VRAM after loading the WebUI:

**Generation 1.**
Selecting a working SDXL model.
Enter prompt. Connect the required LoRA (for example, add-detail-xl.safetensor, ~200Mb size) to see how VRAM is loaded during the generation process.
Prompt:
`photo of a 40yo woman `
Steps: 20
Sampler: DPM++ 2M Karras
CFG scale: 7
(from here on all settings will be the same)

VRAM while generation 1 is in progress:
VRAM after generation 1 is complete:

**Generation 2.**
We're taking the LoRA out of the prompt
Prompt:
`photo of a 40yo woman`

VRAM while generation 2 is in progress:
VRAM after generation 2 is complete:
_So far, everything looks normal._

**AND BUG:
Generation 3.**
Returning the same LoRA to prompt:
`photo of a 40yo woman `

VRAM while generation is in progress:
_(VRAM overflowed, shared memory engaged, generation speed slowed down by ten times)_

VRAM after generation is complete:
_(you can see that after generation is complete, VRAM is not freed as it was after generations 1 and 2)_

### What should have happened?

VRAM must not overflow

### What browsers do you use to access the UI ?

Google Chrome

### Sysinfo

[sysinfo-2024-03-06-12-04.json](https://github.com/lllyasviel/stable-diffusion-webui-forge/files/14509618/sysinfo-2024-03-06-12-04.json)

### Console logs

```Shell
(no errors in console)
```

### Additional information

_No response_

Start by reproducing the documented three-generation sequence with an SDXL model and add-detail-xl LoRA while monitoring VRAM. Compare behavior when removing the LoRA and when using the Never OOM Integrated tab; done means repeated LoRA use does not overflow VRAM, engage shared memory, or leave memory occupied after generation.

Written by the indexing model from the issue text.