multiprocessing train error

Nobody has claimed this yet.

WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 3765 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 3766 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -7) local_rank: 2 (pid: 3767) of binary: /usr/bin/python3
Traceback (most recent call last):
File "/usr/local/bin/torchrun", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/run.py", line 794, in main
run(args)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

train run arges

torchrun --nnodes=1 --nproc_per_node=3 --master_port=20001 train/train.py \
--model_name_or_path /data/model/vicuna/vicuna-7b \
--data_path playground/data/dummy.json \
--bf16 True \
--output_dir /data/app/output \
--num_train_epochs 3 \
--per_device_train_batch_size 4 \
--per_device_eval_batch_size 4 \
--gradient_accumulation_steps 8 \
--evaluation_strategy "no" \
--save_strategy "steps" \
--save_steps 1200 \
--save_total_limit 10 \
--learning_rate 2e-5 \
--weight_decay 0. \
--warmup_ratio 0.03 \
--lr_scheduler_type "cosine" \
--logging_steps 1 \
--logging_dir "/data/app/output" \
--fsdp "full_shard auto_wrap" \
--fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' \
--tf32 True \
--model_max_length 2048 \
--gradient_checkpointing True \
--lazy_preprocess False

Start by reproducing the failure with the provided torchrun command and inspect train/train.py together with the reported ChildFailedError for local_rank 2 and exit code -7. Done means identifying the environment or training configuration that causes the multiprocessing failure and documenting or correcting the issue.

Written by the indexing model from the issue text.