CUDA out of memory, See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Hi, I was not able to run train_db.py. It seemed because CUDA is out of memory. Do I need to set PYTORCH_CUDA_ALLOC_CONF? If so, where should I set?
Also, do I need to empty cache so that it is possible I can train? If so, where should I add code?
Hope someone can help me.

` gradient ccumulation steps / 勾配を合計するステップ数 = 1
total optimization steps / 学習ステップ数: 1600
steps: 0%| | 0/1600 [00:00
train(args)
File "/home/y/sd-scripts/train_db.py", line 321, in train
noise_pred = unet(noisy_latents, timesteps, encoder_hidden_states).sample

File "/home/y/anaconda3/envs/sdscripts/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)

File "/home/y/anaconda3/envs/sdscripts/lib/python3.10/site-packages/accelerate/utils/operations.py", line 490, in __call__
return convert_to_fp32(self.model_forward(*args, **kwargs))

File "/home/y/anaconda3/envs/sdscripts/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
return func(*args, **kwargs)

File "/home/yiggu/anaconda3/envs/sdscripts/lib/python3.10/site-packages/diffusers/models/unet_2d_condition.py", line 407, in forward
sample = upsample_block(
File "/home/y/anaconda3/envs/sdscripts/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)

File "/home/y/anaconda3/envs/sdscripts/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py", line 1198, in forward
hidden_states = torch.utils.checkpoint.checkpoint(

File "/home/y/anaconda3/envs/sdscripts/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 249, in checkpoint
return CheckpointFunction.apply(function, preserve, *args)
File "/home/y/anaconda3/envs/sdscripts/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 107, in forward
outputs = run_function(*args)

File "/home/y/anaconda3/envs/sdscripts/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py", line 1191, in custom_forward

return module(*inputs, return_dict=return_dict)
File "/home/y/anaconda3/envs/sdscripts/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)

File "/home/y/anaconda3/envs/sdscripts/lib/python3.10/site-packages/diffusers/models/attention.py", line 216, in forward
hidden_states = block(hidden_states, context=encoder_hidden_states, timestep=timestep)

File "/home/y/anaconda3/envs/sdscripts/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)

File "/home/y/anaconda3/envs/sdscripts/lib/python3.10/site-packages/diffusers/models/attention.py", line 494, in forward
hidden_states = self.ff(self.norm3(hidden_states)) + hidden_states

File "/home/y/anaconda3/envs/sdscripts/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)

File "/home/y/anaconda3/envs/sdscripts/lib/python3.10/site-packages/diffusers/models/attention.py", line 709, in forward
hidden_states = module(hidden_states)

File "/home/y/anaconda3/envs/sdscripts/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)

File "/home/y/anaconda3/envs/sdscripts/lib/python3.10/site-packages/diffusers/models/attention.py", line 756, in forward
return hidden_states * self.gelu(gate)

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 80.00 MiB (GPU 0; 7.80 GiB total capacity; 6.19 GiB already allocated; 42.81 MiB free; 6.38 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
steps: 0%| | 0/1600 [00:00
sys.exit(main())
File "/home/y/anaconda3/envs/sdscripts/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 45, in main
args.func(args)

File "/home/y/anaconda3/envs/sdscripts/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1104, in launch_command
simple_launcher(args)

File "/home/y/anaconda3/envs/sdscripts/lib/python3.10/site-packages/accelerate/commands/launch.py", line 567, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
`

Start with train_db.py around line 321, where the traceback fails during the UNet call, and review the reported PYTORCH_CUDA_ALLOC_CONF message. Re-run train_db.py under the reported 7.80 GiB GPU conditions and determine whether configuration or code changes are appropriate; done means the cause and a verified resolution are documented.

Written by the indexing model from the issue text.