OutOfMemoryError: CUDA out of memory.

硬件环境：RTX A5000(24GB) * 5
内存：210GB
模型：moss-moon-003-base

训练报错，提示：

OutOfMemoryError: CUDA out of memory. Tried to allocate 3.80 GiB (GPU 0; 23.69 GiB total capacity; 17.46 GiB already allocated; 850.56 MiB free; 22.16 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting 
max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

请问是参数设置，有什么问题吗？

num_machines=1
num_processes=5
machine_rank=0

accelerate launch \
	--config_file ./configs/sft.yaml \
	--num_processes $num_processes \
	--num_machines $num_machines \
	--machine_rank $machine_rank \
	--deepspeed_multinode_launcher standard finetune_moss.py \
	--model_name_or_path /root/autodl-tmp/moss/fnlp/moss-moon-003-base \
	--data_dir ./sft_data \
	--output_dir ./ckpts/moss-moon-003-sft \
	--log_dir ./train_logs/moss-moon-003-sft \
	--n_epochs 2 \
	--train_bsz_per_gpu 1 \
	--eval_bsz_per_gpu 1 \
	--learning_rate 0.000015 \
	--eval_step 200 \
	--save_step 2000

Start by reviewing configs/sft.yaml and finetune_moss.py, then compare their batch, model, and distributed-training settings with the five-GPU launch command. Reproduce the failure on the stated hardware if possible; done means identifying the configuration or allocation condition responsible for the CUDA out-of-memory error and documenting a verified adjustment.