The backward calculation results of DCN in some cases are inconsistent with the DCN of mmcv.

🐛 Describe the bug

When I used torchvision.ops.deform_conv2d to build DCNv2, I found that the calculation result of backward was inconsistent with mmcv's DCNv2.
As far as I know, DCNv2 can be constructed through torchvision's deform_conv2d interface as follows:
(refer to https://github.com/open-mmlab/mmcv/blob/master/mmcv/ops/modulated_deform_conv.py)

import torch
import torch.nn as nn
from torchvision.ops import deform_conv2d as tv_deform_conv2d

class ModulatedDeformConv2d_tv(nn.modules.Module):
    def __init__(self,
                 in_channels: int,
                 out_channels: int,
                 kernel_size: Union[int, Tuple[int]],
                 stride: int = 1,
                 padding: int = 0,
                 dilation: int = 1,
                 groups: int = 1,
                 deform_groups: int = 1,
                 bias: Union[bool, str] = True):
        super(ModulatedDeformConv2d_tv, self).__init__()
        self.in_channels  = in_channels
        self.out_channels = out_channels
        self.kernel_size  = _pair(kernel_size)
        self.stride       = _pair(stride)
        self.padding      = _pair(padding)
        self.dilation     = _pair(dilation)
        self.groups   = groups
        self.deform_groups = deform_groups
        self.weight = nn.Parameter(torch.Tensor(
            out_channels, in_channels, *self.kernel_size))
        if bias:
            self.bias = nn.Parameter(torch.Tensor(out_channels))
        else:
            self.register_parameter('bias', None)
        self.conv_offset = nn.Conv2d(
            self.in_channels,
            self.deform_groups * 3 * self.kernel_size[0] * self.kernel_size[1],
            kernel_size=self.kernel_size,
            stride=self.stride,
            padding=self.padding,
            bias=True)
        self.init_weights()
    def init_weights(self):
        n = self.in_channels
        for k in self.kernel_size:
            n *= k
        stdv = 1. / math.sqrt(n)
        self.weight.data.uniform_(-stdv, stdv)
        if self.bias is not None:
            self.bias.data.zero_()
        self.conv_offset.weight.data.zero_()
        self.conv_offset.bias.data.zero_()
    def forward(self, x):
        out = self.conv_offset(x)
        o1, o2, mask = torch.chunk(out, 3, dim=1)
        offset = torch.cat((o1, o2), dim=1)
        mask = torch.sigmoid(mask)
        return tv_deform_conv2d(x, offset, self.weight,
                                bias=self.bias, stride=self.stride,
                                padding=self.padding, dilation=self.dilation,
                                mask=mask)

Next, I tested it in mmcv's test case. Test case location:
https://github.com/open-mmlab/mmcv/blob/master/tests/test_ops/test_modulated_deform_conv.py

I tested with ModulatedDeformConv2d_tv as described above in place of ModulatedDeformConv2dPack in test_modulated_deform_conv.py.

dcn = ModulatedDeformConv2dPack_tv(1, 1, kernel_size=(2, 2), stride=1, padding=1, deform_groups=1, bias=False)

As a result, there is an error in the calculation of dcn.conv_offset.weight.grad and dcn.conv_offset.bias.grad.

I checked the cpu implementation of torchvision's DCN and found that it was the same paper as mmcv, and it seems that the implementation of tv refers to openmmlab. https://github.com/pytorch/vision/blob/main/torchvision/csrc/ops/cpu/deform_conv2d_kernel.cpp#L67

But both are in https://github.com/pytorch/vision/blob/main/torchvision/csrc/ops/cpu/deform_conv2d_kernel.cpp#L384 and https://github.com/pytorch/vision/blob/main/torchvision/csrc/ops/cpu/deform_conv2d_kernel.cpp#L500 .The processing logic seems to be different from mmcv.
The implementation version of mmcv can refer to :
https://github.com/open-mmlab/mmcv/blob/master/mmcv/ops/csrc/pytorch/cpu/modulated_deform_conv.cpp#L294

Finally, I modified the deform_conv2d_kernel.cpp of torchvision according to the version of mmcv, and the test case passed. It seems that in the case of index=-1, the logic of the two is different.

Therefore, I would like to ask the developers of torchvision, since the implementation of openmmlab is referenced and the reference papers are consistent, is it reasonable that the two frameworks are not aligned in this scenario.

Attached here are modifications to torchvision to align mmcv.

diff --git a/torchvision/csrc/ops/cpu/deform_conv2d_kernel.cpp b/torchvision/csrc/ops/cpu/deform_conv2d_kernel.cpp
index b1d15a1..7b3ca6b 100644
--- a/torchvision/csrc/ops/cpu/deform_conv2d_kernel.cpp
+++ b/torchvision/csrc/ops/cpu/deform_conv2d_kernel.cpp
@@ -388,6 +388,9 @@ scalar_t get_coordinate_weight(
     scalar_t y,
     scalar_t x,
     bool is_y_direction) {
+  if (y <= -1 || y >= height || x <= -1 || x >= width){
+    return 0;
+  }
   int y_l = floor(y);
   int x_l = floor(x);
   int y_h = y_l + 1;
@@ -499,6 +502,10 @@ void deformable_col2im_coord_kernel(
       scalar_t y = (out_y * stride_h - pad_h) + i * dilation_h + offset_h;
       scalar_t x = (out_x * stride_w - pad_w) + j * dilation_w + offset_w;
 
+      if (y <= -1 || x <= -1 || y >= height || x >= width)
+      {
+        x = y = -2;
+      }
       const scalar_t weight =
           get_coordinate_weight(im_ptr, height, width, y, x, is_y_direction);
       grad_offset_val += mask_value * weight * col_ptr[col_pos];

Versions

The version I use is

python==3.6.9
torch==1.9.0
torchvision==0.10.0
mmcv-full==1.6.2

Start in torchvision/csrc/ops/cpu/deform_conv2d_kernel.cpp, especially get_coordinate_weight and deformable_col2im_coord_kernel, and compare their boundary handling with mmcv's modulated_deform_conv.cpp. Reproduce the test_modulated_deform_conv.py case and verify that conv_offset.weight.grad and conv_offset.bias.grad match mmcv for out-of-bounds coordinates.