Issue with DeepSpeed Inference - Multiple Processes for Model Loading and Memory Allocation

Issue Description:
When using DeepSpeed for inference with 2 V100 GPUs, I observed that each GPU requires a separate process for model loading, resulting in loading the model into CPU memory twice before being allocated to the GPUs. After the loading is complete, the model is then sent to the GPUs for memory allocation. Surprisingly, during memory allocation on the GPUs, the model is not duplicated, and the inference runs successfully with both GPUs.

However, when I tried to scale up the setup to 4 machines with 8 GPUs while loading the "bloom 176b-8BIT" model, I encountered a problem. On each machine, every GPU needs to load the model using "bfloat16," requiring approximately 352GB of memory per process on CPU RAM. Since each machine has two GPUs, DeepSpeed starts two processes for model loading, resulting in a requirement of 352GB*2 of GPU memory for model loading. But my machines do not have enough memory to handle this.

Question:
Is it normal for DeepSpeed to create a separate process for each GPU when running inference on a single machine with multiple GPUs? It appears that the model is independently loaded into memory for each GPU, which becomes problematic when the number of GPUs increases. How can I handle this situation effectively, considering the memory constraints?