Performance degradation after using DeepSpeed (UniLM/layoutlmv2)

Hi,

We are trying DeepSpeed with UniLM using FUNSD example that is provided here - https://github.com/microsoft/unilm/tree/master/layoutlmv2

It takes 18-ish minutes to finish training without using DeepSpeed in a 16GB Single GPU machine.
When I use DeepSpeed with the below mentioned configuration, it took 30+ minutes in the same machine. I tried to add optimizer, scheduler with different learning rates etc. I tried different bucket sizes as well. But that did not make any difference.

Can anyone help with this?

Config:
{
"zero_optimization": {
"stage": 2,
"offload_optimizer": {
"device": "cpu",
"pin_memory": true
},
"allgather_partitions": true,
"allgather_bucket_size": 2e8,
"reduce_scatter": true,
"reduce_bucket_size": 2e8,
"overlap_comm": true,
"contiguous_gradients": true
}
}

Versions:
Pytorch: 1.8.0+cu111
DeepSpeed - 5.3
Transformers - 4.5.3 (version that UniLM was bundled with)

Start with the UniLM/layoutlmv2 FUNSD example linked in the report and reproduce training with and without the supplied ZeRO Stage 2 configuration. Compare the two runs and investigate the listed optimizer offload, bucket, and communication settings; done means the cause of the slower DeepSpeed run is identified and a verified configuration or limitation is documented.