Issue with memory filling up with flux dev

Hello, I have an issue where my Dedicated GPU Memory, keeps filling up to almost max, about 23 out of 24 gb vram.
It doesn't seem to matter what I put the gpu weight at, whether it be at 23 gb, 16gb, or 8gb. Same thing happens every time.

Once my Dedicated GPU Memory is almost full, it starts trying to generate an image giving an error about low vram.

Here I am trying it with about 20 gb of vram:

Environment vars changed: {'stream': False, 'inference_memory': 4622.0, 'pin_shared_memory': False}
[GPU Setting] You will use 81.18% GPU memory (19941.00 MB) to load weights, and use 18.82% GPU memory (4622.00 MB) to do matrix computation.
Loading Model: {'checkpoint_info': {'filename': 'D:\\Downloads\\webui_forge_cu124_torch24\\webui\\models\\Stable-diffusion\\Quality_1-000175.safetensors', 'hash': '0a0de032'}, 'additional_modules': ['D:\\Downloads\\webui_forge_cu124_torch24\\webui\\models\\VAE\\ae.safetensors', 'D:\\Downloads\\webui_forge_cu124_torch24\\webui\\models\\text_encoder\\clip_l.safetensors', 'D:\\Downloads\\webui_forge_cu124_torch24\\webui\\models\\text_encoder\\t5xxl_enconly.safetensors'], 'unet_storage_dtype': None}
[Unload] Trying to free all memory for cuda:0 with 0 models keep loaded ... Done.
StateDict Keys: {'transformer': 780, 'vae': 244, 'text_encoder': 196, 'text_encoder_2': 220, 'ignore': 0}
Using Detected T5 Data Type: torch.float8_e4m3fn
Working with z of shape (1, 16, 32, 32) = 16384 dimensions.
K-Model Created: {'storage_dtype': torch.bfloat16, 'computation_dtype': torch.bfloat16}
Model loaded in 67.3s (unload existing model: 0.2s, forge model load: 67.1s).
Skipping unconditional conditioning when CFG = 1. Negative Prompts are ignored.
[Unload] Trying to free 11321.54 MB for cuda:0 with 0 models keep loaded ... Done.
[Memory Management] Target: JointTextEncoder, Free GPU: 15.33 MB, Model Require: 5153.49 MB, Previously Loaded: 0.00 MB, Inference Require: 4622.00 MB, Remaining: -9760.17 MB, CPU Swap Loaded (blocked method): 5152.73 MB, GPU Loaded: 73.14 MB
Moving model(s) has taken 0.09 seconds
Distilled CFG Scale: 3.5
[Unload] Trying to free 11999.54 MB for cuda:0 with 0 models keep loaded ... Current free memory is 383.22 MB ... Unload model JointTextEncoder Done.
[Memory Management] Target: KModel, Free GPU: 73.99 MB, Model Require: 0.00 MB, Previously Loaded: 22700.13 MB, Inference Require: 4622.00 MB, Remaining: -4548.01 MB, CPU Swap Loaded (blocked method): 126.00 MB, GPU Loaded: 22574.13 MB
Moving model(s) has taken 0.26 seconds
  0%|                                                                                           | 0/20 [00:00<?, ?it/s]

----------------------
[Low GPU VRAM Warning] Your current GPU free memory is 67.09 MB for this diffusion iteration.
[Low GPU VRAM Warning] This number is lower than the safe value of 1536.00 MB.
[Low GPU VRAM Warning] If you continue, you may cause NVIDIA GPU performance degradation for this diffusion process, and the speed may be extremely slow (about 10x slower).
[Low GPU VRAM Warning] To solve the problem, you can set the 'GPU Weights' (on the top of page) to a lower value.
[Low GPU VRAM Warning] If you cannot find 'GPU Weights', you can click the 'all' option in the 'UI' area on the left-top corner of the webpage.
[Low GPU VRAM Warning] If you want to take the risk of NVIDIA GPU fallback and test the 10x slower speed, you can (but are highly not recommended to) add '--disable-gpu-warning' to CMD flags to remove this warning.
----------------------

And here with about 8gb of vram:

[GPU Setting] You will use 33.00% GPU memory (8107.00 MB) to load weights, and use 67.00% GPU memory (16456.00 MB) to do matrix computation.
Loading Model: {'checkpoint_info': {'filename': 'D:\\Downloads\\webui_forge_cu124_torch24\\webui\\models\\Stable-diffusion\\Quality_1-000175.safetensors', 'hash': '0a0de032'}, 'additional_modules': ['D:\\Downloads\\webui_forge_cu124_torch24\\webui\\models\\VAE\\ae.safetensors', 'D:\\Downloads\\webui_forge_cu124_torch24\\webui\\models\\text_encoder\\clip_l.safetensors', 'D:\\Downloads\\webui_forge_cu124_torch24\\webui\\models\\text_encoder\\t5xxl_enconly.safetensors'], 'unet_storage_dtype': None}
[Unload] Trying to free all memory for cuda:0 with 0 models keep loaded ... Done.
StateDict Keys: {'transformer': 780, 'vae': 244, 'text_encoder': 196, 'text_encoder_2': 220, 'ignore': 0}
Using Detected T5 Data Type: torch.float8_e4m3fn
Working with z of shape (1, 16, 32, 32) = 16384 dimensions.
K-Model Created: {'storage_dtype': torch.bfloat16, 'computation_dtype': torch.bfloat16}
Model loaded in 105.0s (unload existing model: 0.2s, forge model load: 104.8s).
Skipping unconditional conditioning when CFG = 1. Negative Prompts are ignored.
[Unload] Trying to free 23155.54 MB for cuda:0 with 0 models keep loaded ... Done.
[Memory Management] Target: JointTextEncoder, Free GPU: 15.33 MB, Model Require: 5153.49 MB, Previously Loaded: 0.00 MB, Inference Require: 16456.00 MB, Remaining: -21594.17 MB, CPU Swap Loaded (blocked method): 5152.73 MB, GPU Loaded: 73.14 MB
Moving model(s) has taken 0.09 seconds
Distilled CFG Scale: 3.5
[Unload] Trying to free 23833.54 MB for cuda:0 with 0 models keep loaded ... Current free memory is 383.22 MB ... Unload model JointTextEncoder Done.
[Memory Management] Target: KModel, Free GPU: 73.99 MB, Model Require: 0.00 MB, Previously Loaded: 22700.13 MB, Inference Require: 16456.00 MB, Remaining: -16382.01 MB, CPU Swap Loaded (blocked method): 126.00 MB, GPU Loaded: 22574.13 MB
Moving model(s) has taken 0.28 seconds
  0%|                                                                                           | 0/20 [00:00<?, ?it/s]

----------------------
[Low GPU VRAM Warning] Your current GPU free memory is 67.09 MB for this diffusion iteration.
[Low GPU VRAM Warning] This number is lower than the safe value of 1536.00 MB.
[Low GPU VRAM Warning] If you continue, you may cause NVIDIA GPU performance degradation for this diffusion process, and the speed may be extremely slow (about 10x slower).
[Low GPU VRAM Warning] To solve the problem, you can set the 'GPU Weights' (on the top of page) to a lower value.
[Low GPU VRAM Warning] If you cannot find 'GPU Weights', you can click the 'all' option in the 'UI' area on the left-top corner of the webpage.
[Low GPU VRAM Warning] If you want to take the risk of NVIDIA GPU fallback and test the 10x slower speed, you can (but are highly not recommended to) add '--disable-gpu-warning' to CMD flags to remove this warning.
----------------------

Any help is appreciated!

No source file or test is named. Start by reproducing the reported behavior with the GPU Weights settings shown and inspect the memory-management logs around model loading and inference. Done means explaining why the configured weight values produce nearly identical GPU usage and adding a verified regression check or documented resolution.