BUG: Error with distributed training

Using PyTorch 1.2 with Apex. The code runs just fine on 1 GPU. But, distributed on 4 GPUs is failing.

Output:

$ python -m torch.distributed.launch --nproc_per_node=4 main_amp.py -a resnet50 --b 224 --workers 4 --opt-level O3 data

Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.

opt_level = O3
keep_batchnorm_fp32 = None <class 'NoneType'>
loss_scale = None <class 'NoneType'>
opt_level = O3

CUDNN VERSION: 7602
opt_level = O3
keep_batchnorm_fp32 = None <class 'NoneType'>

loss_scale = None <class 'NoneType'>
keep_batchnorm_fp32 = None <class 'NoneType'>
loss_scale = None <class 'NoneType'>
opt_level = O3
keep_batchnorm_fp32 = None <class 'NoneType'>

CUDNN VERSION: 7602
loss_scale = None <class 'NoneType'>

CUDNN VERSION: 7602

CUDNN VERSION: 7602

=> creating model 'resnet50'
=> creating model 'resnet50'
=> creating model 'resnet50'
=> creating model 'resnet50'
Selected optimization level O3: Pure FP16 training.
Defaults for this optimization level are:
enabled : True
opt_level : O3
cast_model_type : torch.float16
patch_torch_functions : False
keep_batchnorm_fp32 : False
master_weights : False
loss_scale : 1.0
Processing user overrides (additional kwargs that are not None)...
After processing overrides, optimization options are:
enabled : True
opt_level : O3
cast_model_type : torch.float16
patch_torch_functions : False
keep_batchnorm_fp32 : False
master_weights : False
loss_scale : 1.0
Traceback (most recent call last):
File "/mnt/nfs/nfsshare/user_homes/may/anaconda3/envs/apex/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/mnt/nfs/nfsshare/user_homes/may/anaconda3/envs/apex/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/mnt/nfs/nfsshare/user_homes/may/anaconda3/envs/apex/lib/python3.6/site-packages/torch/distributed/launch.py", line 246, in
main()
File "/mnt/nfs/nfsshare/user_homes/may/anaconda3/envs/apex/lib/python3.6/site-packages/torch/distributed/launch.py", line 242, in main
cmd=cmd)
subprocess.CalledProcessError: Command '['/mnt/nfs/nfsshare/user_homes/may/anaconda3/envs/apex/bin/python', '-u', 'main_amp.py', '--local_rank=3', '-a', 'resnet50', '--b', '224', '--workers', '4', '--opt-level', 'O3', 'data']' died with <Signals.SIGSEGV: 11>.

Start by reproducing the command in the issue with PyTorch 1.2, Apex, four GPUs, and main_amp.py using optimization level O3. Inspect the torch.distributed.launch traceback and the main_amp.py execution path around distributed startup. Done means identifying the cause of the SIGSEGV and confirming that distributed training completes successfully.