[REQUEST] Support multiple models using deepspeed

I'm frustrated when I am trying to implement PPO using deepspeed, which needs to run actor, critic and reward model at the same time.

It seems that deepspeed cannot support running multiple models at the same time well. Error always happens like this:
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate xxx MiB

I don't know if there's any functionalities that can support this kind of scenarios. For example, can we specify the gpus each model are using so that we can avoid GPU memory allocation issues by using different GPUs for different models?

Thanks!

Start by reviewing DeepSpeed's support for running multiple models concurrently in the PPO scenario described: actor, critic, and reward model. Investigate how GPU assignment and CUDA memory allocation are handled, then determine the intended behavior for placing separate models on different GPUs. Done should include a defined multi-model workflow that avoids the reported out-of-memory failures.