Does SyncBN support mixed precision training with --opt-level="O2" or "O1" , or --opt-level="O3" with --keep-batchnorm-fp32=True?

Environment:

Framework: PyTorch
Framework version: 1.2
Apex version: 0.1
CUDA version: 10.1
Python version: 3.6.8
OS and version: Ubuntu 16.04
GCC version: 7.4.0

Question:
I'm working with the apex used for mixed training. When I does't use the SyncBN, it can always work well under --opt-level="O1" or "O2" or "O3" with --keep-batchnorm-fp32=True. However, when I convert the BN to SyncBN by model = apex.parallel.convert_syncbn_model(model), it doesn't work.

1. SyncBN with --opt-level="O1"

After processing overrides, optimization options are:
enabled                : True
opt_level              : O1
cast_model_type        : None
patch_torch_functions  : True
keep_batchnorm_fp32    : None
master_weights         : None
loss_scale             : dynamic

Warning:  multi_tensor_applier fused unscale kernel is unavailable, possibly because apex was installed without --cuda_ext --cpp_ext. Using Python fallback. 

It raised error as the following:

File "/workspace/pyroom/runner/runner.py", line 293, in run
    loss, y_pred = self.batch_update(x, y)
File "/workspace/pyroom/runner/runner.py", line 513, in batch_update
    scaled_loss.backward()
File "/opt/conda/lib/python3.6/site-packages/torch/tensor.py", line 118, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/opt/conda/lib/python3.6/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Function SyncBatchnormFunctionBackward returned an invalid gradient at index 0 - expected type torch.cuda.HalfTensor but got torch.cuda.FloatTensor

2. SyncBN with --opt-level="O2"

After processing overrides, optimization options are:
enabled                : True
opt_level              : O2
cast_model_type        : torch.float16
patch_torch_functions  : False
keep_batchnorm_fp32    : True
master_weights         : True
loss_scale             : dynamic
Warning:  multi_tensor_applier fused unscale kernel is unavailable, possibly because apex was installed without --cuda_ext --cpp_ext. Using Python fallback.  

When forward passing with BN layer, it raised error as the following:

File "/workspace/pyroom/model/encoder/resnetEncoder.py", line 222, in forward
    x01 = self.bn1(x01)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/apex/parallel/sync_batchnorm.py", line 109, in forward
    (1 - self.momentum) * self.running_mean
RuntimeError: expected device cuda:0 and dtype Float but got device cuda:0 and dtype Half

3. SyncBN with --opt-level="O3" --keep-batchnorm-fp32=True

After processing overrides, optimization options are:
enabled                : True
opt_level              : O3
cast_model_type        : torch.float16
patch_torch_functions  : False
keep_batchnorm_fp32    : True
master_weights         : False
loss_scale             : 1.0
Warning:  multi_tensor_applier fused unscale kernel is unavailable, possibly because apex was installed without --cuda_ext --cpp_ext. Using Python fallback.  

When forward passing with BN layer, it raised error as the following:

File "/workspace/pyroom/model/encoder/resnetEncoder.py", line 222, in forward
    x01 = self.bn1(x01)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/apex/parallel/sync_batchnorm.py", line 109, in forward
    (1 - self.momentum) * self.running_mean
RuntimeError: expected device cuda:0 and dtype Float but got device cuda:0 and dtype Half

I wonder if SyncBN supports mixed precision training. Is there any kind advice?

Start by reproducing the three reported configurations with apex.parallel.convert_syncbn_model(model), then inspect apex/parallel/sync_batchnorm.py around line 109 and the reported backward path. Compare the tensor dtypes and devices for SyncBN inputs, running statistics, and gradients; done means the supported configurations work without dtype errors, or the limitation is clearly documented.