[BUG] DeepSpeed-inference 2GPU performance is lower than 1GPU

Describe the bug
DeepSpeed-inference 2GPU performance is lower than 1GPU on Bloomz 7.1B

To Reproduce
run the flowing code using two V100 or A100 GPUs.
with this command:
deepspeed --num_gpus 2 --master_port 60000 test.py
`from transformers import AutoModelForCausalLM, AutoTokenizer
import deepspeed

os.environ['CUDA_VISIBLE_DEVICES'] = "0,1"
checkpoint = "bigscience/bloomz-7b1"

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint, torch_dtype="auto", device_map="auto")

model = deepspeed.init_inference(
model=model,
mp_size=2,
dtype=torch.float16,
replace_method="auto",
replace_with_kernel_inject=True,
)

inputs = tokenizer.encode("Translate to English: Je t’aime.", return_tensors="pt").to("cuda")
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))`

The performance of two gpu is lower than one gpu.
The main reason is all_reduce time is too long. The reason of all_reduce too long is because there are many space during the kernel execution, as the picture shows:
the all_reduce time of GPU0 is much longer than GPU1 all_reduce time, because there is long spaces during GPU1 op execution.

Expected behavior
2 GPU inference show be faster than 1 GPU.

• OS:
• Ubuntu 18.04
  -2 A100 or V100 GPU
• Python 3.8

Launcher context
deepspeed --num_gpus 2 --master_port 60000 test.py
Docker context
nvcr.io/nvidia/pytorch:22.09-py3

Start with the test.py reproduction described in the issue and run it with the stated DeepSpeed launcher on two V100 or A100 GPUs. Profile the all_reduce and kernel execution timing to compare one- and two-GPU runs. Done means the two-GPU inference performance improves over one GPU and the reported all_reduce delay is resolved or explained.