RuntimeError: cuda runtime error (74) : misaligned address at /pytorch/aten/src/THC/THCTensorCopy.cu:84

THCudaCheck FAIL file=/pytorch/aten/src/THC/THCTensorCopy.cu line=84 error=74 : misaligned address
Traceback (most recent call last):
File "full_main.py", line 394, in
full_main()
File "full_main.py", line 184, in full_main
loss = train(train_loader, model, criterion, optimizer, epoch, log_training,args.fp16)
File "full_main.py", line 232, in train
output = model(input_var)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(input, kwargs)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/container.py", line 91, in forward
input = module(input)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(input, kwargs)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 124, in forward
return self.gather(outputs, self.output_device)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 136, in gather
return gather(outputs, output_device, dim=self.dim)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 67, in gather
return gather_map(outputs)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 54, in gather_map
return Gather.apply(target_device, dim, outputs)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/_functions.py", line 65, in forward
return comm.gather(inputs, ctx.dim, ctx.target_device)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/cuda/comm.py", line 160, in gather
return torch._C._gather(tensors, dim, destination)
RuntimeError: cuda runtime error (74) : misaligned address at /pytorch/aten/src/THC/THCTensorCopy.cu:84
terminate called after throwing an instance of 'at::Error'
what(): CUDA error: invalid device pointer (CudaCachingDeleter at /pytorch/aten/src/THC/THCCachingAllocator.cpp:498)
frame #0: THStorage_free + 0x44 (0x7ff6e3f710d4 in /home/work/anaconda3/lib/python3.6/site-packages/torch/lib/libcaffe2.so)
frame #1: THTensor_free + 0x2f (0x7ff6e40107df in /home/work/anaconda3/lib/python3.6/site-packages/torch/lib/libcaffe2.so)
frame #2: at::CUDAHalfTensor::~CUDAHalfTensor() + 0x9 (0x7ff64c972ab9 in /home/work/anaconda3/lib/python3.6/site-packages/torch/lib/libcaffe2_gpu.so)
frame #3: torch::autograd::generated::CudnnConvolutionBackward::~CudnnConvolutionBackward() + 0x5d (0x7ff6ebc06afd in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #4: torch::autograd::deleteFunction(torch::autograd::Function) + 0x47 (0x7ff6eb9f8257 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #5: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 0x45 (0x7ff6eb624ea5 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #6: torch::autograd::Function::~Function() + 0xfe (0x7ff6eb6f2f4e in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #7: torch::autograd::generated::CudnnBatchNormBackward::~CudnnBatchNormBackward() + 0x74 (0x7ff6ebc06474 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #8: torch::autograd::deleteFunction(torch::autograd::Function) + 0x47 (0x7ff6eb9f8257 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #9: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 0x45 (0x7ff6eb624ea5 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #10: torch::autograd::Function::~Function() + 0xfe (0x7ff6eb6f2f4e in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #11: torch::autograd::generated::ThresholdBackward1::~ThresholdBackward1() + 0x66 (0x7ff6ebc05cb6 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #12: torch::autograd::deleteFunction(torch::autograd::Function) + 0x47 (0x7ff6eb9f8257 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #13: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 0x45 (0x7ff6eb624ea5 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #14: torch::autograd::Function::~Function() + 0xfe (0x7ff6eb6f2f4e in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #15: torch::autograd::generated::MaxPool2DWithIndicesBackward::~MaxPool2DWithIndicesBackward() + 0x88 (0x7ff6ebc06fa8 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #16: torch::autograd::deleteFunction(torch::autograd::Function) + 0x47 (0x7ff6eb9f8257 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #17: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 0x45 (0x7ff6eb624ea5 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #18: torch::autograd::Function::~Function() + 0xfe (0x7ff6eb6f2f4e in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #19: torch::autograd::generated::CudnnConvolutionBackward::~CudnnConvolutionBackward() + 0x73 (0x7ff6ebc06b13 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #20: torch::autograd::deleteFunction(torch::autograd::Function) + 0x47 (0x7ff6eb9f8257 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #21: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 0x45 (0x7ff6eb624ea5 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #22: torch::autograd::Function::~Function() + 0xfe (0x7ff6eb6f2f4e in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #23: torch::autograd::generated::CudnnBatchNormBackward::~CudnnBatchNormBackward() + 0x74 (0x7ff6ebc06474 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #24: torch::autograd::deleteFunction(torch::autograd::Function) + 0x47 (0x7ff6eb9f8257 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #25: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 0x45 (0x7ff6eb624ea5 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #26: torch::autograd::Function::~Function() + 0xfe (0x7ff6eb6f2f4e in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #27: torch::autograd::generated::ThresholdBackward1::~ThresholdBackward1() + 0x66 (0x7ff6ebc05cb6 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #28: torch::autograd::deleteFunction(torch::autograd::Function) + 0x47 (0x7ff6eb9f8257 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #29: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 0x45 (0x7ff6eb624ea5 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #30: torch::autograd::Function::~Function() + 0xfe (0x7ff6eb6f2f4e in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #31: torch::autograd::generated::CudnnConvolutionBackward::~CudnnConvolutionBackward() + 0x73 (0x7ff6ebc06b13 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #32: torch::autograd::deleteFunction(torch::autograd::Function*) + 0x47 (0x7ff6eb9f8257 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #33: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 0x45 (0x7ff6eb624ea5 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #34: torch::autograd::Function::~Function() + 0xfe (0x7ff6eb6f2f4e in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #35: torch::autograd::generated::CudnnBatchNormBackward::~CudnnBatchNormBackward() + 0x74 (0x7ff6ebc06474 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #36: torch::autograd::deleteFunction(torch::autograd::Function*) + 0x47 (0x7ff6eb9f8257 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #37: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 0x45 (0x7ff6eb624ea5 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #38: torch::autograd::Function::~Function() + 0xfe (0x7ff6eb6f2f4e in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #39: torch::autograd::generated::ThresholdBackward1::~ThresholdBackward1() + 0x66 (0x7ff6ebc05cb6 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #40: torch::autograd::deleteFunction(torch::autograd::Function*) + 0x47 (0x7ff6eb9f8257 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #41: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 0x45 (0x7ff6eb624ea5 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #42: torch::autograd::Function::~Function() + 0xfe (0x7ff6eb6f2f4e in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #43: torch::autograd::generated::MaxPool2DWithIndicesBackward::~MaxPool2DWithIndicesBackward() + 0x88 (0x7ff6ebc06fa8 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #44: torch::autograd::deleteFunction(torch::autograd::Function*) + 0x47 (0x7ff6eb9f8257 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #45: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 0x45 (0x7ff6eb624ea5 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #46: torch::autograd::Function::~Function() + 0xfe (0x7ff6eb6f2f4e in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #47: torch::autograd::generated::AvgPool2DBackward::~AvgPool2DBackward() + 0x67 (0x7ff6ebc065d7 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #48: torch::autograd::deleteFunction(torch::autograd::Function*) + 0x47 (0x7ff6eb9f8257 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #49: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 0x45 (0x7ff6eb624ea5 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #50: torch::autograd::Function::~Function() + 0xfe (0x7ff6eb6f2f4e in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #51: torch::autograd::generated::CudnnConvolutionBackward::~CudnnConvolutionBackward() + 0x73 (0x7ff6ebc06b13 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #52: torch::autograd::deleteFunction(torch::autograd::Function*) + 0x47 (0x7ff6eb9f8257 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #53: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 0x45 (0x7ff6eb624ea5 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #54: torch::autograd::Function::~Function() + 0xfe (0x7ff6eb6f2f4e in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #55: torch::autograd::generated::CudnnBatchNormBackward::~CudnnBatchNormBackward() + 0x74 (0x7ff6ebc06474 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #56: torch::autograd::deleteFunction(torch::autograd::Function*) + 0x47 (0x7ff6eb9f8257 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #57: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 0x45 (0x7ff6eb624ea5 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #58: torch::autograd::Function::~Function() + 0xfe (0x7ff6eb6f2f4e in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #59: torch::autograd::generated::ThresholdBackward1::~ThresholdBackward1() + 0x66 (0x7ff6ebc05cb6 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #60: torch::autograd::deleteFunction(torch::autograd::Function*) + 0x47 (0x7ff6eb9f8257 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #61: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 0x45 (0x7ff6eb624ea5 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #62: torch::autograd::Function::~Function() + 0xfe (0x7ff6eb6f2f4e in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)
frame #63: torch::autograd::generated::CatBackward::~CatBackward() + 0x72 (0x7ff6ebc02132 in /home/work/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)

Aborted

Start with full_main.py, especially train() around line 232 and the fp16/DataParallel path. Reproduce the failure and inspect PyTorch's gather path in torch.nn.parallel.data_parallel and torch.nn.parallel.scatter_gather, alongside the reported THC/THCTensorCopy.cu location. Done means the same training flow no longer raises the misaligned-address or invalid-device-pointer errors.