ComfyUI leaking RAM (not VRAM) after every generation with SDXL 0.9

I tried using two separate workflows and I've encountered the issue anyways.
When looking at my RAM (not VRAM) during the generations, it's always rising, but never goes down. Even with 16GB + 10GB of swap memory, the UI would use it all in 2 or 3 generations.
I never ran out of VRAM.
The programs just crashes and says "Killed", after it used app all of the memory available

Args:
--use-split-cross-attention
--disable-xformers
--dont-upcast-attention

Specs:
CPU -> Ryzen 5 5500
RAM -> 16GB
GPU -> RX 6750XT
OS -> Ubuntu 22.04.02 LTS