RuntimeError: CUDA error: out of memory after crash

I was working to train a new voice model, but something caused my system to completely crash at the 149th epoch, and ever since, when I start it up, I can't train any models because I get a "RuntimeError: CUDA error: out of memory" at the start when I try.

Here is the configuration that was working earlier today:

I've tried setting my batch count to 1 and I was playing around with toggling pitch guidance, but I always get the same error. I've also updated my Nvidia driver, and restarted my computer:

Also here's the full error that I see:
Process Process-1:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/local/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/arashi/git/Retrieval-based-Voice-Conversion-WebUI/train_nsf_sim_cache_sid_load_pretrain.py", line 228, in run
train_and_evaluate(
File "/home/arashi/git/Retrieval-based-Voice-Conversion-WebUI/train_nsf_sim_cache_sid_load_pretrain.py", line 441, in train_and_evaluate
scaler.scale(loss_disc).backward()
File "/home/arashi/.local/lib/python3.10/site-packages/torch/_tensor.py", line 487, in backward
torch.autograd.backward(
File "/home/arashi/.local/lib/python3.10/site-packages/torch/autograd/init.py", line 200, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Has anyone dealt with this issue before that has a solution? Considering that I've tried a batch size of 1 and still got the memory error, I think that my problem is something other than actually not having enough memory.

Thanks

Start by reproducing the reported training failure and read train_nsf_sim_cache_sid_load_pretrain.py around train_and_evaluate and line 441, where the traceback reaches backward(). Compare the reported configuration with the failing run. Done means identifying a reproducible cause and verifying a fix or documenting the remaining limitation.