Having to manually recast output or target (e.g. "expected Tensor but got HalfTensor")

Start with the AMP initialization and loss calculation shown in the issue, then compare their behavior with examples/imagenet/main_amp.py and the AMP documentation linked in the report. Reproduce the mixed-precision error and determine whether target casting is expected; done means the behavior and documentation are consistent without unexplained NaNs.

"With Amp, it isn't necessary to manually convert data to half."
(^from line 263, https://github.com/NVIDIA/apex/blob/master/examples/imagenet/main_amp.py)

"Users should not manually cast their model or data to .half(), regardless of what opt_level or properties are chosen."
(^ from https://nvidia.github.io/apex/amp.html)

I'm getting and error if I leave the targets uncast. Here's a minimal 'schematic' representation of what my code is doing, based on calculating log-cosh error:

model = MyModel()
model.to(device)
optimizer = torch.optim.Adam(list(model.parameters()), lr=lr_sched[0], weight_decay=0)
model, optimizer = amp.initialize(model, optimizer, opt_level="O3")

for x, y_target in my_dataloader:
    x, y_target = x.to(device), y_target.to(device)
    y_pred = model.forward(x)
    loss = torch.mean( torch.log( torch.cosh(y_target - y_pred) ))
    optimizer.zero_grad()
    with amp.scale_loss(loss, optimizer) as scaled_loss:
         scaled_loss.backward()
    torch.nn.utils.clip_grad_norm_(amp.master_params(optimizer), max_norm=1., norm_type=1)
    model.clip_grad_norm_()   
    optimizer.step()

I'm finding that I get a Runtime error,...

RuntimeError: expected type torch.cuda.FloatTensor but got torch.cuda.HalfTensor
unless I manually convert the "y_target" to half, as in...

loss = torch.mean( torch.log( torch.cosh(y_target.half() - y_pred) ))

...this runs but then I get NaNs. Alternatively, manually casting y_pred as float in the loss calculation also runs but also produces NaNs.

The need to manually recast targets seems to be in contradiction to the docs and ImageNet example.

Can someone clarify this? Thanks.

(I see other Issues posted related to this but they seem to involve FusedAdam or BatchNorm or FP16_Optimizer, none of which I'm using.)

PS- The only thing that runs and produces ordinary numbers for me so far, is to revert back to the non-amp-related code.