RuntimeError: reduce failed to synchronize: device-side assert triggered

It's the first time to try Apex in pytorch, several minor problems I've met were solved when I run my code. But when I think it was successful after running several minutes normally，the error below occurred.

/opt/conda/conda-bld/pytorch_1556653183467/work/aten/src/THCUNN/BCECriterion.cu:42: Acctype bce_functor<Dtype, Acctype>::operator()(Tuple) [with Tuple = thrust::detail::tuple_of_iterator_references<thrust::device_reference, thrust::device_reference, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type>, Dtype = float, Acctype = float]: block: [8,0,0], thread: [94,0,0] Assertion input >= 0. && input <= 1. failed.
/opt/conda/conda-bld/pytorch_1556653183467/work/aten/src/THCUNN/BCECriterion.cu:42: Acctype bce_functor<Dtype, Acctype>::operator()(Tuple) [with Tuple = thrust::detail::tuple_of_iterator_references<thrust::device_reference, thrust::device_reference, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type>, Dtype = float, Acctype = float]: block: [8,0,0], thread: [95,0,0] Assertion input >= 0. && input <= 1. failed.
/opt/conda/conda-bld/pytorch_1556653183467/work/aten/src/THCUNN/BCECriterion.cu:42: Acctype bce_functor<Dtype, Acctype>::operator()(Tuple) [with Tuple = thrust::detail::tuple_of_iterator_references<thrust::device_reference, thrust::device_reference, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type>, Dtype = float, Acctype = float]: block: [8,0,0], thread: [96,0,0] Assertion input >= 0. && input <= 1. failed.

Traceback (most recent call last):
a = self.bce_loss(y_true, y_pred)
File "/home/sensetime/anaconda2/envs/pytorch1.1.0-py3-gdal/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/sensetime/anaconda2/envs/pytorch1.1.0-py3-gdal/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 512, in forward
return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)
File "/home/sensetime/anaconda2/envs/pytorch1.1.0-py3-gdal/lib/python3.6/site-packages/apex/amp/wrap.py", line 129, in wrapper
return orig_fn(*args, **kwargs)
File "/home/sensetime/anaconda2/envs/pytorch1.1.0-py3-gdal/lib/python3.6/site-packages/torch/nn/functional.py", line 2113, in binary_cross_entropy
input, target, weight, reduction_enum)
RuntimeError: reduce failed to synchronize: device-side assert triggered

How to solve them?

Start with the BCECriterion.cu assertion shown in the error and the binary_cross_entropy call in torch/nn/functional.py. Check how Apex's amp/wrap.py reaches that loss and determine why the input violates the reported range. Done means identifying whether the failure belongs to Apex or the calling code and documenting a reproducible resolution.