NCCL error in: ../torch/lib/c10d/../c10d/NCCLUtils.hpp:29, unhandled cuda error

My script works with --op-level O0 and O1, but not with the other options. The following is the error log. Any solutions foor this issue?

Defaults for this optimization level are:
enabled                : True
opt_level              : O2
cast_model_type        : torch.float16
patch_torch_functions  : False
keep_batchnorm_fp32    : True
master_weights         : True
loss_scale             : dynamic
Processing user overrides (additional kwargs that are not None)...
After processing overrides, optimization options are:
enabled                : True
opt_level              : O2
cast_model_type        : torch.float16
patch_torch_functions  : False
keep_batchnorm_fp32    : True
master_weights         : True
loss_scale             : dynamic
Learning rate: 0.1
THCudaCheck FAIL file=../aten/src/THC/THCCachingHostAllocator.cpp line=278 error=74 : misaligned address
Traceback (most recent call last):
  File "run_search.py", line 103, in <module>
    main(get_args())
  File "run_search.py", line 59, in main
    loss = searcher.train(epoch)
  File "/mnt/efs/PLNAS/trainer.py", line 220, in train
    output = self.model(input, gates)
  File "/home/ubuntu/anaconda3/envs/pytorchs_py363/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorchs_py363/lib/python3.6/site-packages/apex/parallel/distributed.py", line 476, in forward
    result = self.module(*inputs, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorchs_py363/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorchs_py363/lib/python3.6/site-packages/apex/amp/_initialize.py", line 204, in new_fwd
    **applier(kwargs, input_caster))
  File "/mnt/efs/PLNAS/proxyless_nas/nas_modules.py", line 207, in forward
    x = self.first_conv(x)
  File "/home/ubuntu/anaconda3/envs/pytorchs_py363/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/efs/PLNAS/proxyless_nas/layers.py", line 95, in forward
    x = self.weight_call(x)
  File "/mnt/efs/PLNAS/proxyless_nas/layers.py", line 184, in weight_call
    x = self.conv(x)
  File "/home/ubuntu/anaconda3/envs/pytorchs_py363/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorchs_py363/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 343, in forward
    return self.conv2d_forward(input, self.weight)
  File "/home/ubuntu/anaconda3/envs/pytorchs_py363/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 340, in conv2d_forward
    self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR
terminate called after throwing an instance of 'std::runtime_error'
  what():  NCCL error in: ../torch/lib/c10d/../c10d/NCCLUtils.hpp:29, unhandled cuda error```

The following is the output of pip freeze.

```absl-py==0.7.1
apex==0.1
certifi==2019.6.16
cffi==1.12.3
future==0.17.1
grpcio==1.22.0
Markdown==3.1.1
mkl-fft==1.0.12
mkl-random==1.0.2
numpy==1.16.4
Pillow==6.1.0
protobuf==3.9.1
pycparser==2.19
PyYAML==5.1.1
six==1.12.0
tb-nightly==1.15.0a20190806
torch==1.3.0a0+5ec1c29
torchvision==0.5.0a0+8635be9
typing==3.6.4
Werkzeug==0.15.5```

print(torch.backends.cudnn.version()) gives 7602, however cat /usr/local/cuda/include/cudnn.h | grep CUDNN_MAJOR -A 2 gives 7.4.1.

Using Cuda 10.

Start by reproducing run_search.py with --op-level O0, O1, and O2, then inspect trainer.py, nas_modules.py, and layers.py at the stack-trace locations. Compare the reported PyTorch, Apex, CUDA, and cuDNN versions while isolating the misaligned-address and cuDNN failures; done means a confirmed cause and a reproducible resolution.