[BUG] Deepspeed multi-GPU Inference

Describe the bug

I'm using transformers.Trainer with deepspeed. During training, Zero 2 is adopted. For evaluation, I just want to accelerate with multi-GPU inference like in normal DDP, while deepspeed raises ValueError: "ZeRO inference only makes sense with ZeRO Stage 3 - please adjust your config".
I find that in evaluation it works, but direct calling "trainer.predict" after initialize the trainer would cause the above error.

I understand the error, that means optimizer and scheduler of ZeRO 2 is not optimized during train, but is there an approach that allows multiple-GPUs inference given configure of ZeRO 2?