CUDA error: an illegal memory access was encountered

I'm not sure how to address this, could you please give me a suggestion:

Defaults for this optimization level are:
enabled                : True
opt_level              : O1
cast_model_type        : None
patch_torch_functions  : True
keep_batchnorm_fp32    : None
master_weights         : None
loss_scale             : dynamic
Processing user overrides (additional kwargs that are not None)...
After processing overrides, optimization options are:
enabled                : True
opt_level              : O1
cast_model_type        : None
patch_torch_functions  : True
keep_batchnorm_fp32    : None
master_weights         : None
loss_scale             : dynamic
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCCachingHostAllocator.cpp line=278 error=700 : an illegal memory access was encountered
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCCachingHostAllocator.cpp line=278 error=700 : an illegal memory access was encountered
terminate called after throwing an instance of 'std::runtime_error'
  what():  NCCL error in: /pytorch/torch/lib/c10d/../c10d/NCCLUtils.hpp:84, unhandled cuda error
terminate called after throwing an instance of 'std::runtime_error'
  what():  NCCL error in: /pytorch/torch/lib/c10d/../c10d/NCCLUtils.hpp:84, unhandled cuda error
Traceback (most recent call last):
  File "build_baseline_model.py", line 265, in <module>
    __IPYTHON__
NameError: name '__IPYTHON__' is not defined

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "build_baseline_model.py", line 268, in <module>
    main()
  File "build_baseline_model.py", line 260, in main
    torch.multiprocessing.spawn(build_model, nprocs=args.number_gpus, args=(args,))
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn
    while not spawn_context.join():
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 118, in join
    raise Exception(msg)
Exception:

-- Process 1 terminated with the following error:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "/root/src/kaggle/lyft3d/build_baseline_model.py", line 90, in build_model
    model_trainer.train(gpu=gpu, epochs=args.epochs, train_iterator=train_iterator, validation_iterator=validation_iterator, criterion=criterion, clip=args.clip, hyperparameters=args, checkpoint_timestamp=args.resume_checkpoint, scheduler=scheduler)
  File "/root/src/kaggle/lyft3d/training/model_trainer.py", line 162, in train
    scaled_loss.backward()
  File "/opt/conda/lib/python3.7/contextlib.py", line 119, in __exit__
    next(self.gen)
  File "/opt/conda/lib/python3.7/site-packages/apex/amp/handle.py", line 127, in scale_loss
    should_skip = False if delay_overflow_check else loss_scaler.update_scale()
  File "/opt/conda/lib/python3.7/site-packages/apex/amp/scaler.py", line 200, in update_scale
    self._has_overflow = self._overflow_buf.item()
RuntimeError: CUDA error: an illegal memory access was encountered

Start with build_baseline_model.py around main() and build_model(), then inspect training/model_trainer.py at the scaled_loss.backward() call and the Apex amp scaler traceback. Reproduce the failure with the reported multi-GPU mixed-precision setup and capture the PyTorch, Apex, CUDA, and NCCL versions. Done means isolating a reproducible cause or providing a minimal case that distinguishes Apex from the underlying CUDA/NCCL failure.