RuntimeError: Magnitude of gradient is bad: nan

Hi,

The output of my softmax contains a few zeros due to key masking. I apply a query mask after the softmax using cmult. The problem is that I get the error that magnitude of gradient is bad: nan. I have narrowed it down to be due to the cmult (if commented the training works fine).

The explanation that I have come up with is that since there are a few values in the output of softmax which are zeros, there are also a few values in the query mask which are zero and occur at corresponding positions. Thus resulting in 0*0. The code works fine if not training, but while training I get the run-time error. Does this seem an appropriate enough reason for the gradients becoming nan?

Also then what to replace the 0's in the query mask with to avoid this problem?

Start by reproducing the training-time failure with softmax key masking, a query mask, and cmult enabled, then compare it with the working case where cmult is removed. Trace the masked softmax and cmult inputs through gradient computation; done means the cause of the NaN gradient is established and a validated masking behavior is identified.

Written by the indexing model from the issue text.