Fine-Tuning Crashes for no reason when Eight GPU cards are used.

Dear Developers at LMFlow:

I have been using LMFlow for a long time and the experience is great !

But recently, after cloning the latest LMFlow and use it to Fine-Tune my model, I encountered some expected issue.

Specifically, when I use all 8 of my NVIDIA-A100 cards, the fine-tuning program crashes without indicating any error. However, when I use only 6 / 7 cards, things goes well.

Beliw is the output of the program:

[2024-05-08 14:29:20,246] [WARNING] [comm.py:152:init_deepspeed_backend] NCCL backend in DeepSpeed not yet implemented
[2024-05-08 14:29:20,246] [INFO] [comm.py:616:init_distributed] cdb=None
[2024-05-08 14:29:20,398] [WARNING] [comm.py:152:init_deepspeed_backend] NCCL backend in DeepSpeed not yet implemented
[2024-05-08 14:29:20,399] [INFO] [comm.py:616:init_distributed] cdb=None
[2024-05-08 14:29:20,458] [WARNING] [comm.py:152:init_deepspeed_backend] NCCL backend in DeepSpeed not yet implemented
[2024-05-08 14:29:20,458] [INFO] [comm.py:616:init_distributed] cdb=None
[2024-05-08 14:29:20,499] [WARNING] [comm.py:152:init_deepspeed_backend] NCCL backend in DeepSpeed not yet implemented
[2024-05-08 14:29:20,500] [INFO] [comm.py:616:init_distributed] cdb=None
[2024-05-08 14:29:20,537] [WARNING] [comm.py:152:init_deepspeed_backend] NCCL backend in DeepSpeed not yet implemented
[2024-05-08 14:29:20,537] [INFO] [comm.py:616:init_distributed] cdb=None
[2024-05-08 14:29:20,538] [INFO] [comm.py:643:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
[2024-05-08 14:29:20,593] [WARNING] [comm.py:152:init_deepspeed_backend] NCCL backend in DeepSpeed not yet implemented
[2024-05-08 14:29:20,593] [INFO] [comm.py:616:init_distributed] cdb=None
[2024-05-08 14:29:20,634] [WARNING] [comm.py:152:init_deepspeed_backend] NCCL backend in DeepSpeed not yet implemented
[2024-05-08 14:29:20,634] [INFO] [comm.py:616:init_distributed] cdb=None
[2024-05-08 14:29:20,650] [WARNING] [comm.py:152:init_deepspeed_backend] NCCL backend in DeepSpeed not yet implemented
[2024-05-08 14:29:20,650] [INFO] [comm.py:616:init_distributed] cdb=None
05/08/2024 14:29:21 - WARNING - lmflow.pipeline.finetuner - Process rank: 5, device: cuda:5, n_gpu: 1,distributed training: True, 16-bits training: True
05/08/2024 14:29:21 - WARNING - lmflow.pipeline.finetuner - Process rank: 6, device: cuda:6, n_gpu: 1,distributed training: True, 16-bits training: True
05/08/2024 14:29:21 - WARNING - lmflow.pipeline.finetuner - Process rank: 4, device: cuda:4, n_gpu: 1,distributed training: True, 16-bits training: True
05/08/2024 14:29:21 - WARNING - lmflow.pipeline.finetuner - Process rank: 0, device: cuda:0, n_gpu: 1,distributed training: True, 16-bits training: True
05/08/2024 14:29:21 - WARNING - lmflow.pipeline.finetuner - Process rank: 1, device: cuda:1, n_gpu: 1,distributed training: True, 16-bits training: True
05/08/2024 14:29:22 - WARNING - lmflow.pipeline.finetuner - Process rank: 3, device: cuda:3, n_gpu: 1,distributed training: True, 16-bits training: True
05/08/2024 14:29:22 - WARNING - lmflow.pipeline.finetuner - Process rank: 7, device: cuda:7, n_gpu: 1,distributed training: True, 16-bits training: True
05/08/2024 14:29:22 - WARNING - lmflow.pipeline.finetuner - Process rank: 2, device: cuda:2, n_gpu: 1,distributed training: True, 16-bits training: True
[2024-05-08 14:32:25,933] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 3906770
[2024-05-08 14:32:25,972] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 3906771
[2024-05-08 14:32:31,375] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 3906772
[2024-05-08 14:32:35,169] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 3906773
[2024-05-08 14:32:38,199] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 3906774
[2024-05-08 14:32:41,567] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 3906775
[2024-05-08 14:32:45,223] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 3906776
[2024-05-08 14:32:48,678] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 3906777

[2024-05-08 14:32:53,189] [ERROR] [launch.py:321:sigkill_handler] ['miniconda3/envs/lmflow/bin/python', '-u', 'examples/finetune.py', '--local_rank=7', '--model_name_or_path', 'meta-llama/Meta-Llama-3-8B', '--dataset_path', '/data', '--output_dir', '/model', '--overwrite_output_dir', '--num_train_epochs', '1', '--learning_rate', '1e-5', '--block_size', '512', '--per_device_train_batch_size', '32', '--deepspeed', 'configs/ds_config_zero3.json', '--fp16', '--run_name', 'chinese-llama3', '--validation_split_percentage', '20', '--logging_steps', '20', '--do_train', '--ddp_timeout', '72000', '--save_steps', '5000', '--use_flash_attention', 'True', '--dataloader_num_workers', '8'] exits with return code = -9

I am pretty sure I use the correct way to specify the GPU cards to use by setting DeepSpeed Arguments:
deepspeed_args="--master_port=11012 --include localhost:0,1,2,3,4,5,6,7"

As I have never excountered this problem with the elder version, after several times of experiment with errors, I come here to seek for helps.
I am not sure if it is the problem on the LMFlow side / on my side.

Thanks for your help ~

Start with examples/finetune.py and configs/ds_config_zero3.json, reproducing the reported run with eight GPUs and then six or seven. Compare the DeepSpeed, distributed-training, and process-termination output, including the provided command-line settings. Done means identifying whether the failure is in LMFlow or the surrounding environment and documenting a reproducible diagnosis or actionable fix.