[BUG]Multinode train: node1 OOM but node2 still calculating, but without generating any checkpoint or gaining steps

Describe the bug
A clear and concise description of what the bug is.

To Reproduce

Just make one node OOM

Expected behavior
Quit all node after one node OOM

ds_report output
Please run ds_report to give us details about your setup.

--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-devel package with yum
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
fused_adam ............. [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.1
 [WARNING]  using untested triton version (2.1.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/root/space/conda_envs/hubc12/lib/python3.10/site-packages/torch']
torch version .................... 2.1.1
deepspeed install path ........... ['/root/space/conda_envs/hubc12/lib/python3.10/site-packages/deepspeed']
deepspeed info ................... 0.10.2, unknown, unknown
torch cuda version ............... 12.1
torch hip version ................ None
nvcc version ..................... 12.1
deepspeed wheel compiled w. ...... torch 2.1, cuda 12.1
shared memory (/dev/shm) size .... 78.33 GB

Screenshots
If applicable, add screenshots to help explain your problem.

System info (please complete the following information):

• OS: centos 7
• GPU count and types 2v1002 4v1002
• Interconnects (if applicable) [e.g., two machines connected with 100 Gbps IB]
• Python version 3.10.13
• Any other relevant info about your setup

Launcher context
Are you launching your experiment with the deepspeed launcher, MPI, or something else?

Docker context
Are you using a specific docker image that you can share?

Additional context
Add any other context about the problem here.

Start by reproducing the multinode training failure using the reported DeepSpeed 0.10.2, PyTorch 2.1.1, and CUDA 12.1 environment, with one node forced to OOM; review the distributed launcher and failure-handling entry points reached by that run. Done means all nodes exit when one node encounters an OOM instead of continuing without checkpoints or progress.