SDXL lora memory leak under linux

When I load an SDXL based lora, my regular ram fills up entirely. Comfy UI manages to get to the sampler node before getting stuck. I gave my pc 40gb of zram (similar to swap) and it filled the entire thing and crashed anyway, leading me to believe this problem to is a memory leak. I have tested the loras I am using in A1111 and they work, so the problem does not seem to be with the loras themself. The problem even occurs with stabilityAI's example SDXL lora. I managed to almost fix it by launching with --force-fp16, which resolves the memory leak but causes the generation of blank black images. Running in CPU mode does not resolve this either.

This is what the terminal looks like prior to freezing:
```
Starting server

To see the GUI go to: http://127.0.0.1:8188
got prompt
model_type EPS
adm 2816
making attention of type 'vanilla-pytorch' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla-pytorch' with 512 in_channels
missing {'cond_stage_model.clip_l.text_projection', 'cond_stage_model.clip_l.logit_scale'}
left over keys: dict_keys(['conditioner.embedders.1.model.transformer.text_model.embeddings.position_ids', 'cond_stage_model.clip_l.transformer.text_model.embeddings.position_ids'])
loading new
loading new
```
Hardware:
Gpu: Nvidia GeForce GTX 1650 mobile (4gb vram)
Cpu: Intel i5-9300H (8 cores)
OS: ArchLinux