ConnectionError in finetuning 7b-v1.3

Nobody has claimed this yet.

I am doing finetune task. I run the script using the following command:

torchrun --nproc_per_node=4 --master_port=20001 fastchat/train/train_mem.py \
--model_name_or_path vicuna-7b-v1.3 \
--data_path v1.0/character_expand_dataset_v1.json \
--bf16 True \
--output_dir v1.0/model \
--num_train_epochs 2 \
--per_device_train_batch_size 1 \
--per_device_eval_batch_size 1 \
--gradient_accumulation_steps 4 \
--evaluation_strategy "no" \
--save_strategy "steps" \
--save_steps 1200 \
--save_total_limit 10 \
--learning_rate 2e-5 \
--weight_decay 0. \
--warmup_ratio 0.03 \
--lr_scheduler_type "cosine" \
--logging_steps 1 \
--fsdp "full_shard auto_wrap offload" \
--fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' \
--tf32 True \
--model_max_length 512 \
--gradient_checkpointing True \
--lazy_preprocess True

I got the error in the following:

ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 13944 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 13945 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 2 (pid: 13946) of binary: /opt/conda/bin/python3.9
Traceback (most recent call last):
File "/opt/conda/bin/torchrun", line 8, in
sys.exit(main())
File "/opt/conda/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
File "/opt/conda/lib/python3.9/site-packages/torch/distributed/run.py", line 794, in main
run(args)
File "/opt/conda/lib/python3.9/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/opt/conda/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/opt/conda/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

Start by reproducing the provided torchrun command and inspect fastchat/train/train_mem.py around model loading and distributed startup. Trace the ConnectionError and the failing local rank before the elastic launcher terminates the other processes. Done means identifying a reproducible cause and confirming the finetuning command completes or reports a clear actionable error.

Written by the indexing model from the issue text.