ConnectionError when I finetune the model.

Nobody has claimed this yet.

ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 15087 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 15088 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 15089 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 15090 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 15091 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 15093 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 15094 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 5 (pid: 15092) of binary: /opt/conda/bin/python3.9
Traceback (most recent call last):
File "/opt/conda/bin/torchrun", line 8, in
sys.exit(main())
File "/opt/conda/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
File "/opt/conda/lib/python3.9/site-packages/torch/distributed/run.py", line 794, in main
run(args)
File "/opt/conda/lib/python3.9/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/opt/conda/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/opt/conda/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
fastchat/train/train_mem.py FAILED
------------------------------------------------------------
Failures:

------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2023-07-14_07:43:00
host : node36
rank : 5 (local_rank: 5)
exitcode : 1 (pid: 15092)
error_file:
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

Start with fastchat/train/train_mem.py and the reported torchrun failure, then inspect the PyTorch distributed elastic error documentation linked in the traceback. Reproduce the finetuning command and identify why rank 5 receives the connection reset; done means the finetuning run completes without the ConnectionError.

Written by the indexing model from the issue text.