[BUG] Support for MoE model inference

Describe the bug
Hi -- I'm trying to build an example to demonstrate expert parallelism feature as described here. I'm getting an error when initiating the inference engine with MoE option enabled.

To Reproduce
Here's the code that cause the problem:

tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
model = BertModel.from_pretrained("/mnt/checkpoint/").cuda()
ds_engine = deepspeed.init_inference(model, mp_size=1, dtype=torch.half, replace_method='auto', replace_with_kernel_inject=True, moe=True, moe_experts=6, ep_size=6)
model = ds_engine.module

output = model(**inputs)

ds_report output

--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
sparse_attn ............ [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
utils .................. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/opt/conda/lib/python3.8/site-packages/torch']
torch version .................... 1.11.0a0+b6df043
torch cuda version ............... 11.5
nvcc version ..................... 11.5
deepspeed install path ........... ['/opt/conda/lib/python3.8/site-packages/deepspeed']
deepspeed info ................... 0.6.0+bea701a, bea701a, master
deepspeed wheel compiled w. ...... torch 1.11, cuda 11.5

Screenshots

System info (please complete the following information):

• OS: Ubuntu 20.04.3 LTS
• GPU count and types : one machine with 2 V100
• Interconnects (if applicable)

 root@6d4e5d59f2ce:/mnt# nvidia-smi topo --matrix
	GPU0	GPU1	mlx5_0	mlx5_1	CPU Affinity	NUMA Affinity
GPU0	 X 	SYS	SYS	SYS	0,2,4,6,8,10	0
GPU1	SYS	 X 	NODE	NODE	1,3,5,7,9,11	1
mlx5_0	SYS	NODE	 X 	PIX		
mlx5_1	SYS	NODE	PIX	 X 		

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

• Python version: Python 3.8.12

Launcher context
From scripts

Docker context
nvcr.io/nvidia/pytorch:22.01-py3

Is there an example to properly using MoE inference using DeepSpeed? Thanks.

Start by reproducing the provided model setup with deepspeed.init_inference using moe=True, moe_experts=6, and ep_size=6, then inspect the inference initialization path and the ds_report output. Done means MoE inference works for this configuration or the supported limitations and a working example are documented.