CUDA out of memory

**Describe**
CUDA out of memory.
I'm fine-tuning the llama-2-70B using 3 sets of machines containing 8\*A100s (40GB)=24\*A100(40GB), and this error reported at first seemed like it should be an out-of-memory issue, but a large enough amount of memory has been used in the calculations.

**To Reproduce**
1. pip install xtuner
2. I replaced the model address of the huggingface in llama2_70b_qlora_open_platypus_e1.py with the Llama-2-70b-hf downloaded locally:
```
# model
pretrained_model_name_or_path = '/mnt/model/Llama-2-70b-hf'
# and also the dataset
data_path = '/mnt/model/Open-Platypus'
```
3.
```
Master(A100*8): NPROC_PER_NODE=8 NNODES=3 NODE_RANK=0 PORT=34545 ADDR=192.168.0.6 xtuner train llama2_70b_qlora_open_platypus_e1
(and [A100*8]NODE_RANK=1,[A100*8]NODE_RANK=2)
```

**System info**
- OS: Ubuntu 20.04.6 LTS
- Configured as 3 groups with 8*A100 graphics cards (total of 24 A100-40G graphics cards)
- Python = 3.10

**ERROR record**

```
model = MMDistributedDataParallel(torch.cudatorch.cuda
..torch.cuda File "/mnt/anaconda/envs/xtuner/lib/python3.10/site-packages/mmengine/model/wrappers/distributed.py", line 93, in __init__
OutOfMemoryErrorOutOfMemoryError.: : OutOfMemoryErrorCUDA out of memory. Tried to allocate 2.11 GiB (GPU 6; 39.45 GiB total capacity; 37.14 GiB already allocated; 1.60 GiB free; 37.30 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONFCUDA out of memory. Tried to allocate 2.11 GiB (GPU 4; 39.45 GiB total capacity; 37.14 GiB already allocated; 1.60 GiB free; 37.30 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF:

CUDA out of memory. Tried to allocate 2.11 GiB (GPU 5; 39.45 GiB total capacity; 37.14 GiB already allocated; 1.60 GiB free; 37.30 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
super().__init__(module=module, **kwargs)
File "/mnt/anaconda/envs/xtuner/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 688, in __init__
self._ddp_init_helper(
File "/mnt/anaconda/envs/xtuner/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 825, in _ddp_init_helper
self.reducer = dist.Reducer(
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.11 GiB (GPU 7; 39.45 GiB total capacity; 37.14 GiB already allocated; 1.60 GiB free; 37.30 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 2120836) of binary: /mnt/anaconda/envs/xtuner/bin/python
Traceback (most recent call last):
File "/mnt/anaconda/envs/xtuner/bin/torchrun", line 8, in
sys.exit(main())
File "/mnt/anaconda/envs/xtuner/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
File "/mnt/anaconda/envs/xtuner/lib/python3.10/site-packages/torch/distributed/run.py", line 794, in main
run(args)
File "/mnt/anaconda/envs/xtuner/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/mnt/anaconda/envs/xtuner/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/mnt/anaconda/envs/xtuner/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
/mnt/anaconda/envs/xtuner/lib/python3.10/site-packages/xtuner/tools/train.py FAILED
------------------------------------------------------------
Failures:
[1]:
time : 2023-09-02_01:48:53
host : gzyd29
rank : 1 (local_rank: 1)
exitcode : 1 (pid: 2120837)
error_file:
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
time : 2023-09-02_01:48:53
host : gzyd29
rank : 2 (local_rank: 2)
exitcode : 1 (pid: 2120838)
error_file:
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
time : 2023-09-02_01:48:53
host : gzyd29
rank : 3 (local_rank: 3)
exitcode : 1 (pid: 2120839)
error_file:
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[4]:
time : 2023-09-02_01:48:53
host : gzyd29
rank : 4 (local_rank: 4)
exitcode : 1 (pid: 2120840)
error_file:
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[5]:
time : 2023-09-02_01:48:53
host : gzyd29
rank : 5 (local_rank: 5)
exitcode : 1 (pid: 2120841)
error_file:
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[6]:
time : 2023-09-02_01:48:53
host : gzyd29
rank : 6 (local_rank: 6)
exitcode : 1 (pid: 2120842)
error_file:
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[7]:
time : 2023-09-02_01:48:53
host : gzyd29
rank : 7 (local_rank: 7)
exitcode : 1 (pid: 2120843)
error_file:
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2023-09-02_01:48:53
host : gzyd29
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 2120836)
error_file:
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
```