Expected object of scalar type Float but got scalar type Half with opt_level 02,03

pytorch: 1.3.1
cuda: 10.1
os: windows 10

I'm getting this error on my encoder-decoder with attention network if I use opt_level 02 or 03. 01 is working fine :

cell.repeat(self.T -1, 1, 1).permute(1, 0, 2), input_encoded), dim=2)
RuntimeError: Expected object of scalar type Float but got scalar type Half for sequence element 2 in sequence argument at position #1 'tensors'

this is what I'm using:

[encoder, decoder], [encoder_optimizer, decoder_optimizer] = amp.initialize([encoder, decoder],   [encoder_optimizer, decoder_optimizer], opt_level="O2")


with amp.scale_loss(loss, [encoder_optimizer, decoder_optimizer]) as scaled_loss:
        scaled_loss.backward()

and here is what decoder looks like:

`class Decoder(nn.Module):

def __init__(self, encoder_hidden_size, decoder_hidden_size, history_size):
    super(Decoder, self).__init__()
    self.encoder_hidden_size = encoder_hidden_size
    self.decoder_hidden_size = decoder_hidden_size
    self.T = history_size + 1

    self.attn_layer = nn.Sequential(nn.Linear(2 * decoder_hidden_size + encoder_hidden_size, encoder_hidden_size),
                                    nn.Tanh(), nn.Linear(encoder_hidden_size, 1))
    self.lstm1 = nn.LSTM(1, decoder_hidden_size, 1)
    self.out1 = nn.Linear(encoder_hidden_size + 1, 1)
    self.out2 = nn.Linear(decoder_hidden_size + encoder_hidden_size, 24)

    self.out1.weight.data.normal_()

def forward(self, input_encoded, y_history, target=None):
    hidden = self.init_hidden()
    cell = self.init_hidden()     

    for i in range(self.T - 1):
        x = torch.cat((hidden.repeat(self.T - 1, 1, 1).permute(1, 0, 2),
                        cell.repeat(self.T -1, 1, 1).permute(1, 0, 2), input_encoded), dim=2)
        x = F.softmax(
                self.attn_layer(
                    x.view(-1, 2 * self.decoder_hidden_size + self.encoder_hidden_size)
                ).view(-1, self.T - 1), dim=1)  # (batch_size, T - 1)

        context = torch.bmm(x.unsqueeze(1), input_encoded)[:, 0, :]     # batch_size * encoder_hidden_size
        y_tilde = self.out1(torch.cat((context, y_history[:, i]), dim=1))  # (batch_size, out_size)
        
        # Eqn. 16: LSTM
        self.lstm1.flatten_parameters()
        _, lstm_output = self.lstm1(y_tilde.unsqueeze(0), (hidden, cell))

        hidden = lstm_output[0]  # 1 * batch_size * decoder_hidden_size
        cell = lstm_output[1]  # 1 * batch_size * decoder_hidden_size          

    y_pred = self.out2(torch.cat((hidden[0], context), dim=1))
    return predictions`

Start with the reported amp.initialize(..., opt_level="O2") path and the Decoder.forward torch.cat call that fails under O2/O3. Reproduce with the stated PyTorch 1.3.1, CUDA 10.1, and Windows setup, then trace the Float/Half boundary; done means the responsible behavior is fixed or clearly identified with a regression test or documented limitation.