UI has been slow on low VRAM for a few months

### Prerequisites

- [x] I am running the latest version of ComfyUI
- [x] I have searched existing issues to make sure this isn't a duplicate
- [x] I have tested with all custom nodes disabled ([see how](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled))

### What happened?

In order to preserve as much vram for my discrete GPU, I'm running Comfyui with one screen running off my integrated GPU (integrated on the cpu) with 2Go VRAM. Lately, the UI has become extremely slow (it takes 1-2 seconds to pan, disable/enable node, select node, edit node value etc). I suspect this has to do with VRAM swap on my integrated GPU, because this does not happen when I run on a monitor plugged in my discrete GPU. The UI is so slow, that it becomes hard and frustrating to interact with (displacing nodes is a nightmare, group select as well, doing anything fast and reliably feels almost impossible).

This happens even before any model is loaded, before any task is put in queue.

It is hard to pinpoint when (i.e. what Comfyui version) the change occured, especially since I don't update more than circa once/month. I'd say about 2 months, maybe a bit more.
I was running comfyui-frontend-package version 1.28.8, just upgraded (using pip) to 1.33.3 and it is the same.

Are there settings I could change to mitigate this issue ? I already lowered the max FPS of the UI to 15.

### Steps to Reproduce

1. Run Comfyui off of screen ran by very low end / integrated GPU
2.
3.

### How is this affecting you?

Visual/UI issue only

### ComfyUI Frontend Version

1.33.3

### Browser

Chrome/Chromium

### Console Errors

```javascript

```

### Logs

```shell

```

### Additional Context

_No response_

┆Issue is synchronized with this [Notion page](https://www.notion.so/Issue-6780-UI-has-been-slow-on-low-VRAM-for-a-few-months-2b16d73d36508149b02cfeee8d765927) by [Unito](https://www.unito.io)