ScatterGather exceptions triggered by `distributions.Categorical.log_prob`

When training my model which samples from Categorical distribution I every now and then get this stack trace

/cvlabdata2/home/tyszkiew/PhD/points/algorithm.py in point_distribution(logits)
     23     spatial_dist = Categorical(logits=logits)
     24     candidate_marks = spatial_dist.sample()
---> 25     spatial_logp = spatial_dist.log_prob(candidate_marks)
     26 
     27     survival_logits = torch.gather(

/usr/local/lib/python3.6/dist-packages/torch/distributions/categorical.py in log_prob(self, value)
    114         value, log_pmf = torch.broadcast_tensors(value, self.logits)
    115         value = value[..., :1]
--> 116         return log_pmf.gather(-1, value).squeeze(-1)
    117 
    118     def entropy(self):

RuntimeError: cuda runtime error (710) : device-side assert triggered at /pytorch/aten/src/THC/generic/THCTensorScatterGather.cu:75

This problem seems to be ameliorated by decreasing the magnitude of logits, but is otherwise elusive. I was thinking that normalization fails due to numerical precision and samples go outside of the allowed range but it seems to be handled.

Start with the stack trace in algorithm.py, especially point_distribution, and inspect the referenced torch/distributions/categorical.py log_prob path. Reproduce the intermittent CUDA device-side assertion with the reported Categorical logits and compare behavior as their magnitude changes; done means identifying the failure cause and covering the corrected behavior with a regression test.