OOM using AdamW8bit since recent update

I'm getting OOM errors using adamw8bit on my 16GB 4080 w/32gb system ram now.
It normally doesn't fit in it's vram anyway, but on commit 0005867ba509d2e1a5674b267e8286b561c0ed71 it was able to still train despite spilling over into shared memory, and the difference was night and day from using Adafactor, converging faster with much better results.
Is there any way to prevent it from using enough to OOM?

`accelerate launch --mixed_precision bf16 --num_cpu_threads_per_process 1 flux_train_network.py --pretrained_model_name_or_path "flux1-dev-fp8.safetensors" --clip_l "clip_l.safetensors" --t5xxl "t5xxl_fp16.safetensors" --ae "ae.safetensors" --cache_latents_to_disk --save_model_as safetensors --sdpa --persistent_data_loader_workers --max_data_loader_n_workers 2 --seed 42 --gradient_checkpointing --mixed_precision bf16 --save_precision bf16 --network_module networks.lora_flux --network_dim 16 --optimizer_type adamw8bit --learning_rate 1e-4 --fp8_base --max_train_epochs 400 --save_every_n_epochs 2 --dataset_config model.toml --output_dir outputs --output_name model --timestep_sampling shift --discrete_flow_shift 3.1582 --model_prediction_type raw --guidance_scale 1.0 --sample_every_n_epochs 1 --sample_prompts prompts.txt --network_args "loraplus_unet_lr_ratio=4" "train_t5xxl=True" "split_qkv=True"`

Start with the supplied accelerate launch command and the flux_train_network.py entry point, then compare the current behavior with commit 0005867ba509d2e1a5674b267e8286b561c0ed71. Reproduce the AdamW8bit run on the stated hardware and identify what changed in memory use; done means the regression is addressed without the reported OOM.

Written by the indexing model from the issue text.