Model parallel: an illegal memory access was encountered

I'm running my model to process really large 3D volumes, so I have to define a model parallel like this:
Class model(....):
def forward(self, x):
#x is on 'cuda:0'
XA=self.A(x)
XB=self.B(XA.to('cuda:1'))
return XB
It runs well using float32, but still I want larger volume size or more channels, so I tried apex and it reported:
RuntimeError: CUDA error: an illegal memory access was encountered
from the scaler.py: self._has_overflow = self._overflow_buf.item()
Any ideas?

Start with the failure reported at scaler.py, specifically self._overflow_buf.item(), and reproduce the described model-parallel run with mixed precision versus float32 on the two CUDA devices. Trace whether the illegal access originates in overflow handling or the preceding model-parallel operation; done means the cause and a verified resolution for this configuration are established.