[BUG] process exits with return code=-6 during training with bf16 optimizer

Describe the bug
We are training a llama2-70b model on 16 A800 GPU servers, and using the following features:

• bf16
• activation checkpoint
• flash attentiion
• zero1

We has successfully run about 800 steps, loss decline normally. And then suddenly processes in one node exits with code = -6.
After about half an hour, all the processes in other nodes exit.
** What exactly causes the error code=-6 ? **

here is the final exit log:

192.169.83.120: [2023-10-13 11:31:28,593] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 249
192.169.83.120: [2023-10-13 11:31:58,630] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 250
192.169.83.120: [2023-10-13 11:31:58,641] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 251
192.169.83.120: [2023-10-13 11:32:28,663] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 252
192.169.83.120: [2023-10-13 11:32:28,663] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 253
192.169.83.120: [2023-10-13 11:32:58,685] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 254
192.169.83.120: [2023-10-13 11:33:28,706] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 255
192.169.83.120: [2023-10-13 11:33:58,758] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 256
192.169.83.120: [2023-10-13 11:34:28,773] [ERROR] [launch.py:321:sigkill_handler] ['/usr/bin/python3.10', '-u', '/mnt/tenant-home_speed/shanghai/gpt/train/main.py', '--local_rank=7', '--micro-batch-size', '1', '--pipeline-model-parallel-size', '16', '--tensor-model-parallel-size', '1', '--num-layers', '80', '--hidden-size', '8192', '--num-attention-heads', '64', '--max-position-embeddings', '4096', '--seq-length', '4096', '--distributed-backend', 'nccl', '--deepspeed', '--tokenizer-type', 'NullTokenizer', '--vocab-size', '32024', '--bf16', '--clip-grad', '1.0', '--checkpoint-activations', '--checkpoint-num-layers', '1', '--use-flash-attn', '--config_file', 'run_bf16_with_job.json'] exits with return code = -6

Expected behavior
we expect training process will continue normally.

System info (please complete the following information):

• OS: Ubuntu 20.04.6 LTS
• CUDA-VERSION: 11.8
• DeepSpeed version: 0.10.2
• GPU count and types: 16 machines with x8 A800s each
• Interconnects (if applicable) : machines connected with 100 Gbps IB
• Python version: 3.10

Launcher context
we launching the experiment with the deepspeed launcher.

Start with the deepspeed launcher entry point shown in the log, especially launch.py:321:sigkill_handler, and the training entry point /mnt/tenant-home_speed/shanghai/gpt/train/main.py. Compare the first node's stderr and launcher output with the later termination messages across the 16 nodes. Done means identifying the cause of return code -6 and confirming that training can continue under the reported configuration.