AttributeError: 'DistributedDataParallel' object has no attribute 'global_steps'

Hi,

I am trying to reproduce the results of ZeRO on GPT2 and BERT pretraining.
I followed this tutorial:
https://github.com/microsoft/DeepSpeedExamples/blob/master/Megatron-LM-v1.1.5-ZeRO3/README.md

1. Dataset that I used:

I followed Aaron Gokaslan and Vanya Cohen's tutorial to download training dataset and DeepSpeedExamples' tutorial to preprocess the data (https://github.com/microsoft/DeepSpeedExamples/blob/master/Megatron-LM-v1.1.5-ZeRO3/README.md#data-preprocessing).
In my test, I just used only a part of the dataset.

2. Script that I used:

I modified the provided script (DeepSpeedExamples/Megatron-LM-v1.1.5-ZeRO3/examples/pretrain_gpt2.sh) accordingly:

#! /bin/bash

# Runs the "345M" parameter model

RANK=0
WORLD_SIZE=1

DATA_PATH=../my-gpt2_text_document
CHECKPOINT_PATH=../pretrain_models/gpt2/release/mp_rank_00/model_optim_rng

LOCAL_RANK=0

python ../pretrain_gpt2.py \
       --num-layers 24 \
       --hidden-size 1024 \
       --num-attention-heads 16 \
       --batch-size 8 \
       --seq-length 1024 \
       --max-position-embeddings 1024 \
       --train-iters 500000 \
       --lr-decay-iters 320000 \
       --save $CHECKPOINT_PATH \
       --load $CHECKPOINT_PATH \
       --data-path $DATA_PATH \
       --vocab-file ../pretrain_models/gpt2/gpt2-vocab.json \
       --merge-file ../pretrain_models/gpt2/gpt2-merges.txt \
       --data-impl mmap \
       --split 949,50,1 \
       --distributed-backend nccl \
       --lr 0.00015 \
       --min-lr 1.0e-5 \
       --lr-decay-style cosine \
       --weight-decay 1e-2 \
       --clip-grad 1.0 \
       --warmup .01 \
       --checkpoint-activations \
       --log-interval 100 \
       --save-interval 10000 \
       --eval-interval 1000 \
       --eval-iters 10 \
       --fp16


set +x