RuntimeError: expected scalar type Half but found Float

What is the best way to figure out where this issue is happening in the graph? The message is also unclear: What exactly is expecting the Half? would be helpful to print the particular node in the graph where this breaks.

Sorry, I don't really have a super clean implementation which can reproduce this.

  File "/private/home/falc/.conda/envs/ddt_debug/lib/python3.7/site-packages/pytorch_lightning/trainer/train_loop_mixin.py", line 194, in optimizer_closure
    model_ref.backward(self.use_amp, closure_loss, optimizer)
  File "/private/home/falc/.conda/envs/ddt_debug/lib/python3.7/site-packages/pytorch_lightning/root_module/hooks.py", line 70, in backward
    scaled_loss.backward()
  File "/private/home/falc/.local/lib/python3.7/site-packages/torch/tensor.py", line 118, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/private/home/falc/.local/lib/python3.7/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: expected scalar type Half but found Float

The piece of code that likely has the issue is here:

    def nce_loss(self, r_src, r_trg, mask_mat):
        '''
        Compute the NCE scores for predicting r_src->r_trg.
        Input:
          r_src    : (batch_size, emb_dim)
          r_trg    : (emb_dim, n_batch * w* h) (ie: nb_feat_vectors x embedding_dim)
          mask_mat : (n_batch_gpu, n_batch)
        Output:
          raw_scores : (n_batch_gpu, n_locs)
          nce_scores : (n_batch_gpu, n_locs)
          lgt_reg    : scalar
        '''

        # RKHS = embedding dim
        batch_size, emb_dim = r_src.size()
        nb_feat_vectors = r_trg.size(1) // batch_size

        # (b, b) -> (b, b, nb_feat_vectors)
        # all zeros with ones in diagonal tensor... (ie: b1 b1 are all 1s, b1 b2 are all zeros)
        mask_pos = mask_mat.unsqueeze(dim=2).expand(-1, -1, nb_feat_vectors).float()

        # negative mask
        # one = torch.ones_like(mask_pos)
        mask_neg = 1. - mask_pos

        # -------------------------------
        # ALL SCORES COMPUTATION
        # compute src->trg raw scores for batch
        # (b, dim) x (dim, nb_feats*b) -> (b, b, nb_feats)
        # vector for each img in batch times all the vectors of all images in batch
        raw_scores = torch.mm(r_src, r_trg)
        raw_scores = raw_scores.reshape(batch_size, batch_size, nb_feat_vectors).float()

        # -----------------------
        # STABILITY TRICKS
        # trick 1: weighted regularization term
        raw_scores = raw_scores / emb_dim**0.5
        lgt_reg = 5e-2 * (raw_scores**2).mean()

        # trick 2: tanh clip
        raw_scores = tanh_clip(raw_scores, clip_val=self.tclip).float()

        '''
        pos_scores includes scores for all the positive samples
        neg_scores includes scores for all the negative samples, with
        scores for positive samples set to the min score (-self.tclip here)
        '''
        # ----------------------
        # EXTRACT POSITIVE SCORES
        # use the index mask to pull all the diagonals which are b1 x b1
        # (batch_size, nb_feat_vectors)
        pos_scores = (mask_pos * raw_scores).sum(dim=1).float()

        # ----------------------
        # EXTRACT NEGATIVE SCORES
        # pull everything except diagonal and apply clipping
        # (batch_size, batch_size, nb_feat_vectors)
        # diagonals have - clip vals. everything else has actual negative stores
        neg_scores = (mask_neg * raw_scores) - (self.tclip * mask_pos)

        # (batch_size, batch_size * nb_feat_vectors) -> (batch_size, batch_size, nb_feat_vectors)
        neg_scores = neg_scores.reshape(batch_size, -1)
        mask_neg = mask_neg.reshape(batch_size, -1)

        # ---------------------
        # STABLE SOFTMAX
        # max for each row of negative samples
        # will use max in safe softmax
        # (n_batch_gpu, 1)
        neg_maxes = torch.max(neg_scores, dim=1, keepdim=True)[0]

        # DENOMINATOR
        # sum over only negative samples (none from the diagonal)
        neg_sumexp = (mask_neg * torch.exp(neg_scores - neg_maxes)).sum(dim=1, keepdim=True)
        all_logsumexp = torch.log(torch.exp(pos_scores - neg_maxes) + neg_sumexp)

        # FULL NCE
        # NUMERATOR
        # compute numerators for the NCE log-softmaxes
        pos_shiftexp = pos_scores - neg_maxes

        nce_scores = pos_shiftexp - all_logsumexp
        nce_scores = -nce_scores.mean().float()

        return nce_scores, lgt_reg

Start with the pasted nce_loss function and the mixed-precision backward path shown in the traceback. Reproduce the Half/Float failure with a minimal graph if possible, then determine what diagnostic context is available at the failing node. Done means the error identifies the operation or graph node and clarifies which value has the unexpected dtype.