[REQUEST] how to enable fp16 when using pipeline method

I set fp16 settings as below:

    "fp16": {
        "enabled": true,
        "fp16_master_weights_and_grads": false,
        "loss_scale": 0,
        "loss_scale_window": 500,
        "hysteresis": 2,
        "min_loss_scale": 1,
        "initial_scale_power": 15
    }

when running engine.train_batch(), I got error below:

  File "/home/xihe/xinhe/ColossalNAS/deep_speed/train.py", line 178, in train_pipe
    loss = engine.train_batch()
  File "/home/xihe/xinhe/deepspeed/DeepSpeed/deepspeed/runtime/pipe/engine.py", line 346, in train_batch
    self._exec_schedule(sched)
  File "/home/xihe/xinhe/deepspeed/DeepSpeed/deepspeed/runtime/pipe/engine.py", line 1376, in _exec_schedule
    self._exec_instr(**cmd.kwargs)
  File "/home/xihe/xinhe/deepspeed/DeepSpeed/deepspeed/runtime/pipe/engine.py", line 658, in _exec_forward_pass
    outputs = super().forward(inputs)
  File "/home/xihe/xinhe/deepspeed/DeepSpeed/deepspeed/utils/nvtx.py", line 11, in wrapped_fn
    ret_val = func(*args, **kwargs)
  File "/home/xihe/xinhe/deepspeed/DeepSpeed/deepspeed/runtime/engine.py", line 1846, in forward
    loss = self.module(*inputs, **kwargs)
  File "/datasets/xihe/miniconda3/envs/colossal/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/xihe/xinhe/deepspeed/DeepSpeed/deepspeed/runtime/pipe/module.py", line 344, in forward
    x = func(forward_input)
  File "/home/xihe/xinhe/deepspeed/DeepSpeed/deepspeed/runtime/pipe/module.py", line 337, in exec_func
    inputs = layer(inputs)
  File "/datasets/xihe/miniconda3/envs/colossal/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/datasets/xihe/miniconda3/envs/colossal/lib/python3.9/site-packages/torch/nn/modules/container.py", line 139, in forward
    input = module(input)
  File "/datasets/xihe/miniconda3/envs/colossal/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/datasets/xihe/miniconda3/envs/colossal/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 457, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/datasets/xihe/miniconda3/envs/colossal/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 453, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.cuda.HalfTensor) should be the same

Start with deepspeed/runtime/pipe/engine.py at train_batch and _exec_forward_pass, then follow the call into deepspeed/runtime/engine.py and deepspeed/runtime/pipe/module.py. Reproduce the reported conv2d dtype mismatch using the supplied fp16 settings and pipeline training path. Done means pipeline training proceeds without the FloatTensor/HalfTensor mismatch under this configuration.