out of memory when finetune Vicuna-7B with 4 x A100 (40GB) or 8 x A100 (40GB)

Nobody has claimed this yet.

I followed readme.md https://github.com/lm-sys/FastChat#fine-tuning-vicuna-7b-with-local-gpus, but it did not work, still out of memory. same issue #407 #605

how can i solve the issue ??

```
OutOfMemoryError: CUDA out of memory. Tried to allocate 98.00 MiB (GPU 2; 39.59 GiB total capacity; 37.33 GiB already allocated; 16.19 MiB free; 37.68 GiB reserved in total by PyTorch) If reserved memory is >>
allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 57249 closing signal SIGTERM
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /gpudisk1/Works/FastChat-alpha/wandb/offline-run-20230626_204546-umbyhvta
wandb: Find logs at: ./wandb/offline-run-20230626_204546-umbyhvta/logs
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 1 (pid: 57250) of binary: /gpudisk1/anaconda3/envs/vicuna-env/bin/python
Traceback (most recent call last):
File "/gpudisk1/anaconda3/envs/vicuna-env/bin/torchrun", line 33, in
sys.exit(load_entry_point('torch==2.0.1', 'console_scripts', 'torchrun')())
File "/gpudisk1/anaconda3/envs/vicuna-env/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
File "/gpudisk1/anaconda3/envs/vicuna-env/lib/python3.10/site-packages/torch/distributed/run.py", line 794, in main
run(args)
File "/gpudisk1/anaconda3/envs/vicuna-env/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/gpudisk1/anaconda3/envs/vicuna-env/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/gpudisk1/anaconda3/envs/vicuna-env/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
fastchat/train/train_mem.py FAILED
------------------------------------------------------------
Failures:
[1]:
time : 2023-06-26_20:45:59
host : VM-0-11-ubuntu
rank : 2 (local_rank: 2)
exitcode : 1 (pid: 57251)
error_file:
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
time : 2023-06-26_20:45:59
host : VM-0-11-ubuntu
rank : 3 (local_rank: 3)
exitcode : 1 (pid: 57252)
error_file:
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
time : 2023-06-26_20:45:59
host : VM-0-11-ubuntu
rank : 4 (local_rank: 4)
exitcode : 1 (pid: 57253)
error_file:
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[4]:
time : 2023-06-26_20:45:59
host : VM-0-11-ubuntu
rank : 5 (local_rank: 5)
exitcode : 1 (pid: 57254)
error_file:
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[5]:
time : 2023-06-26_20:45:59
host : VM-0-11-ubuntu
rank : 6 (local_rank: 6)
exitcode : 1 (pid: 57255)
error_file:
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[6]:
time : 2023-06-26_20:45:59
host : VM-0-11-ubuntu
rank : 7 (local_rank: 7)
exitcode : 1 (pid: 57256)
error_file:
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2023-06-26_20:45:59
host : VM-0-11-ubuntu
rank : 1 (local_rank: 1)
exitcode : 1 (pid: 57250)
error_file:
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
```

Start with the README.md section on fine-tuning Vicuna-7B with local GPUs, then inspect fastchat/train/train_mem.py and the reported PyTorch CUDA out-of-memory trace. Reproduce the failure with the documented setup and determine which configuration or memory condition causes it; done should include a confirmed diagnosis and a documented or tested resolution.

Written by the indexing model from the issue text.