[BUG] zero_to_fp32.py cannot convert the model

Describe the bug
I'm training a model using 4 V100 GPUs, and after training, I'm trying to run the zero_to_fp32.py script to convert my model. But the script throws the following error. Please how can I fix this error?

Processing zero checkpoint './global_stepXXX'
Traceback (most recent call last):
  File "zero_to_fp32.py", line 453, in <module>
    convert_zero_checkpoint_to_fp32_state_dict(args.checkpoint_dir, args.output_file)
  File "zero_to_fp32.py", line 391, in convert_zero_checkpoint_to_fp32_state_dict
    state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag)
  File "zero_to_fp32.py", line 377, in get_fp32_state_dict_from_zero_checkpoint
    return _get_fp32_state_dict_from_zero_checkpoint(ds_checkpoint_dir)
  File "zero_to_fp32.py", line 137, in _get_fp32_state_dict_from_zero_checkpoint
    zero_stage, world_size, param_shapes, fp32_flat_groups = parse_optim_states(optim_files, ds_checkpoint_dir)
  File "zero_to_fp32.py", line 94, in parse_optim_states
    f"Expected {world_size} of '*_optim_states.pt' under '{ds_checkpoint_dir}' but found {total_files} files. "
ValueError: Expected 1 of '*_optim_states.pt' under './global_stepXXX' but found 4 files. Possibly due to an overwrite of an old checkpoint, or a checkpoint didn't get saved by one or more processes.

To Reproduce
Steps to reproduce the behavior:

1. run zero_to_fp32.py script

Expected behavior
A clear and concise description of what you expected to happen.

ds_report output

--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
sparse_attn ............ [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
transformer_inference .. [NO] ....... [OKAY]
utils .................. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/home/user/miniconda3/envs/py3.6/lib/python3.6/site-packages/torch']
torch version .................... 1.10.1
torch cuda version ............... 10.2
nvcc version ..................... 10.2
deepspeed install path ........... ['/home/user/miniconda3/envs/py3.6/lib/python3.6/site-packages/deepspeed']
deepspeed info ................... 0.5.10, unknown, unknown
deepspeed wheel compiled w. ...... torch 1.10, cuda 10.2

System info (please complete the following information):

• OS: Ubuntu 18.04.4 LTS
• GPU count and types: one machines with x4 V100s
• Python version: python 3.6
• Any other relevant info about your setup

Launcher context
Are you launching your experiment with the deepspeed launcher, MPI, or something else?
I use deepspeed launcher

Docker context
Are you using a specific docker image that you can share?
No

Addtional context
The model I am using is DeepSpeedExamples/Megatron-LM-v1.1.5-3D_parallelism and the config file is the same as it.

Start with zero_to_fp32.py, especially parse_optim_states and the failing call chain shown in the traceback. Inspect the checkpoint directory and the deepspeed launcher/config context to determine why the script expects one optimizer-state file but finds four; done means the reported checkpoint can be converted successfully or the failure condition is clearly documented.