Hybrid DDP (bug?)

Hi

I encountered this bug (?) using apex DDP.
The model lives on three GPUs, so-called model parallel.
(The data x goes through GPU-0 and then GPU-1 and GPU-3.)
Then, I use DDP to run two processes, so 6 GPUs in total on a single node (dgx-1).

The error I encountered is
"RuntimeError: Event device 4 does not match recording stream's device 3." for rank-1 and
"RuntimeError: Event device 1 does not match recording stream's device 0."
for rank-0.

If I switch to pytorch's native DDP, no error occurs.
I assume this is some optimization using cuda stream went wrong...
I cannot reproduce the problem using "fake" data on my prototype code.
Some environments: pytorch1.3, cuda10.1, apex 1.0, (all installed using conda)

Here is the full trace:

 16   File "/sdcc/u/yren/.conda/envs/torch13/lib/python3.7/site-packages/torch/tensor.py", line 166, in backward
 15     torch.autograd.backward(self, gradient, retain_graph, create_graph)
 14   File "/sdcc/u/yren/.conda/envs/torch13/lib/python3.7/site-packages/torch/autograd/__init__.py", line 99, in backward
 13     allow_unreachable=True)  # allow_unreachable flag
 12   File "/sdcc/u/yren/.conda/envs/torch13/lib/python3.7/site-packages/apex/parallel/distributed.py", line 400, in allreduc    e_hook
 11     self.comm_ready_buckets(param)
 10   File "/sdcc/u/yren/.conda/envs/torch13/lib/python3.7/site-packages/apex/parallel/distributed.py", line 534, in comm_rea    dy_buckets
  9     self.allreduce_maybe_retain(self.buckets[bucket_idx], bucket_idx)
  8   File "/sdcc/u/yren/.conda/envs/torch13/lib/python3.7/site-packages/apex/parallel/distributed.py", line 479, in allreduc    e_maybe_retain
  7     allreduced = self.allreduce_bucket(bucket, bucket_idx, force_default_stream)
  6   File "/sdcc/u/yren/.conda/envs/torch13/lib/python3.7/site-packages/apex/parallel/distributed.py", line 433, in allreduc    e_bucket
  5     torch.cuda.current_stream().record_event(bucket_event)
  4   File "/sdcc/u/yren/.conda/envs/torch13/lib/python3.7/site-packages/torch/cuda/streams.py", line 67, in record_event
  3     event.record(self)
  2   File "/sdcc/u/yren/.conda/envs/torch13/lib/python3.7/site-packages/torch/cuda/streams.py", line 145, in record
  1     super(Event, self).record(stream)
  0 RuntimeError: Event device 4 does not match recording stream's device 3.

Start in apex/parallel/distributed.py, especially allreduce_bucket and the stream/event handling shown in the traceback. Reproduce the two-process, model-parallel setup if possible and compare it with native DDP; done means the run completes without an event-device mismatch.