Input type (torch.cuda.FloatTensor) and weight type (torch.cuda.HalfTensor) should be the same

Hi,

I receive the error shown below when I try FP16 training (opt_level="03"). When training on opt_level="01", everything seems to be working fine. I have attached a snippet of the code with relevant parts. I believe I have followed your documentation but maybe I am missing something.

Thanks for help

PyTorch: 1.4.0
Cuda: 10.1

Code snippet:
`

...
from apex.fp16_utils import *
from apex import amp, optimizers 
...

model = MyModel()
optimizer = torch.optim.Adam(model.parameters(),lr=0.01, eps=10**-7)

model, optimizer = amp.initialize(model, optimizer, opt_level="O3")

for epoch in range(nb_epoch):

    optimizer = lr_scheduler(optimizer, epoch)

    for i, inputs in enumerate(train_loader):

        inputs = inputs.permute(0, 1, 4, 2, 3)
        inputs = inputs.cuda()
        errors = model(inputs)
        errors = errors.float()
        loc_batch = errors.size(0)
        errors = torch.mm(errors.view(-1, nt), time_loss_weights)
        errors = torch.mm(errors.view(loc_batch, -1), layer_loss_weights)
        errors = torch.mean(errors)
        optimizer.zero_grad()
        with amp.scale_loss(errors, optimizer) as scaled_loss:
            scaled_loss.backward()
            #errors.backward()
        optimizer.step()

'
Error:

Selected optimization level O3:  Pure FP16 training.
Defaults for this optimization level are:
enabled                : True
opt_level              : O3
cast_model_type        : torch.float16
patch_torch_functions  : False
keep_batchnorm_fp32    : False
master_weights         : False
loss_scale             : 1.0
Processing user overrides (additional kwargs that are not None)...
After processing overrides, optimization options are:
enabled                : True
opt_level              : O3
cast_model_type        : torch.float16
patch_torch_functions  : False
keep_batchnorm_fp32    : False
master_weights         : False
loss_scale             : 1.0
Traceback (most recent call last):
  File "train_t_1.py", line 96, in <module>
    errors = model(inputs)
  File "/home/bernard/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/bernard/miniconda3/lib/python3.7/site-packages/apex/amp/_initialize.py", line 197, in new_fwd
    **applier(kwargs, input_caster))
  File "/home/bernard/Projects/PrednetConvLSTMPytorch/PredNetOriginal.py", line 141, in forward
    Rep, Cell = cell(tmp, Cell)
  File "/home/bernard/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/bernard/Projects/PrednetConvLSTMPytorch/ConvLSTMCellPredNet.py", line 41, in forward
    i_t = torch.sigmoid(self.W_i(inputs)) #Bias included in self.W_.. initialization
  File "/home/bernard/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/bernard/miniconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 345, in forward
    return self.conv2d_forward(input, self.weight)
  File "/home/bernard/miniconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 342, in conv2d_forward
    self.padding, self.dilation, self.groups)
RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.cuda.HalfTensor) should be the same

`

Start with the traceback entry point in apex/amp/_initialize.py and the reported model call in PredNetOriginal.py and ConvLSTMCellPredNet.py. Reproduce the failure with PyTorch 1.4.0, CUDA 10.1, and opt_level="O3", then verify that the reported input and weight type mismatch no longer occurs.