the question for bing-bert

I want to pretrain the bert-large with only single V100-32G, to reproduce the result as follow diagram. However,In BERT pre-training tutorial, it has different hyper-parameters and different script, maybe the code have been updated. I try to run ds_train_bert_nvidia_data_bsz64k_seq128.sh(change train_batch_size=320, and train_micro_batch_size_per_gpu=32)in single GPU to instead the methods of tutorail. And add all the optimize method to reduce the memory and imporve the speed.

python ${base_dir}/deepspeed_train.py
--cf ${base_dir}/bert_large_lamb_nvidia_data.json
--max_seq_length 512
--output_dir $OUTPUT_DIR
--print_steps 1
--deepspeed
--deepspeed_transformer_kernel
--stochastic_mode
--gelu_checkpoint
--normalize_invertible
--job_name $JOB_NAME
--deepspeed_config ${base_dir}/deepspeed_bsz32k_lamb_config_seq512.json
--data_path_prefix /workspace/bert
--use_nvidia_dataset
--rewarmup
--lr_schedule "EE"
--attention_dropout_checkpoint
--lr_offset 0.0
--load_training_checkpoint ${CHECKPOINT_BASE_PATH}
--load_checkpoint_id ${CHECKPOINT_EPOCH_NAME} \

The above are the specific script. However, the test result of samples/seconds is about 47. It is smaller than 52 a lot. Can you give me some suggestions about the result？ What's more, I get the same result about megatron，so there are no hardware problems. Thanks a lot!

Start with ds_train_bert_nvidia_data_bsz64k_seq128.sh, deepspeed_train.py, and the referenced BERT JSON configuration files; compare their batch, sequence, and optimization settings with the tutorial. Done means documenting a reproducible explanation for the reported 47 rather than 52 samples per second, or identifying a configuration issue.