[Bug]: Idle VRAM usage with --always-offload-from-vram is way higher than vanilla auto1111 --medvram

### Checklist

- [X] The issue exists after disabling all extensions
- [X] The issue exists on a clean installation of webui
- [ ] The issue is caused by an extension, but I believe it is caused by a bug in the webui
- [X] The issue exists in the current version of the webui
- [X] The issue has not been reported before recently
- [ ] The issue has been reported before but has not been fixed yet

### What happened?

Vanilla webui with --medvram after generating once to offload will use 200,000KB of vram, Forge however (using --always-offload-from-vram) will use 800,000KB and sometimes even grows(? possibly another bug?) up to 1GB when idle.

### Steps to reproduce the problem

1. Launch with --always-offload-from-vram
2. Generate once (Model is only offloaded after generating once for some reason. Different bug I guess, exists on both repos)
3. Check task manager or Operating system equivalent.

### What should have happened?

Really it should try to minimise idle vram usage, I'm not sure what changed. Possibly some optimisation that medvram did that isn't done anymore?

### What browsers do you use to access the UI ?

Google Chrome, Apple Safari, iOS

### Sysinfo

[sysinfo-2024-02-21-05-27.json](https://github.com/lllyasviel/stable-diffusion-webui-forge/files/14354459/sysinfo-2024-02-21-05-27.json)

### Console logs

```Shell
Nothing out of the usual, regular old logs. My local repo is not currently checked out on forge branch to fetch one. But if you need one I can get one for you.
```

### Additional information

_No response_

Reproduce the issue with --always-offload-from-vram, generate once, and measure idle VRAM usage using the task manager or an operating-system equivalent. Compare the result with vanilla webui using --medvram and inspect the attached sysinfo JSON for relevant environment details. Done means idle VRAM is minimized rather than remaining around 800,000KB or growing toward 1GB.

Written by the indexing model from the issue text.