DeformConv2d with zero offsets deviates from Conv2d

🐛 Describe the bug

Hi,

when using torchvision.ops.DeformConv2d with an offset of all zeros, the result derivates from the result that I get from torch.nn.Conv2d. Since the offsets are all zeros, I would expect the outputs to be equal. When having only a few input and output channels the difference is quite small (but still unexpected) but for a larger network the difference can get quite significant.

Here is a minimal example to reproduce the issue:

import torch
import torchvision

torch.backends.cudnn.deterministic = True

C_in, H, W = 1, 3, 3 # size of input
C_out = 1 # number of out channels
device = 'cpu'
torch.manual_seed(1)

num_tries = 100
num_correct = 0

for i in range(num_tries):
    img = torch.rand((1,C_in,H, W)).to(device)

    weight = torch.rand((C_out,C_in,3,3)).to(device) # create weight and bias
    bias = torch.rand((C_out)).to(device)

    conv1 = torch.nn.Conv2d(C_in,C_out,kernel_size = 3, padding=0).to(device)
    conv2 = torchvision.ops.DeformConv2d(C_in, C_out, kernel_size = 3, padding = 0).to(device)

    conv1.weight = torch.nn.parameter.Parameter(weight)
    conv1.bias = torch.nn.parameter.Parameter(bias)

    conv2.weight = torch.nn.parameter.Parameter(weight)
    conv2.bias = torch.nn.parameter.Parameter(bias)
    offsets = torch.zeros((1,18,1,1)).to(device) # H_out, W_out = 1 --> shape is 1,18,1,1

    out1 = conv1(img)
    out2 = conv2(img,offsets)
    
    num_correct += (out1 == out2).all().float()
    
print('Out of %d runs, only %d produced identical results.' % (num_tries, num_correct))

Output:

Out of 100 runs, only 56 produced identical results.

Versions

Collecting environment information...
PyTorch version: 1.10.1
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.10

Python version: 3.7.9 (default, Aug 31 2020, 12:42:55) [GCC 7.3.0] (64-bit runtime)
Python platform: Linux-4.15.0-135-generic-x86_64-with-debian-buster-sid
Is CUDA available: True
CUDA runtime version: 10.2.89
GPU models and configuration: GPU 0: GeForce RTX 3090
Nvidia driver version: 460.39
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.19.1
[pip3] torch==1.10.1
[pip3] torchfile==0.1.0
[pip3] torchvision==0.11.2
[conda] blas 1.0 mkl anaconda
[conda] captum 0.3.1 0 pytorch
[conda] cudatoolkit 11.3.1 h2bc3f7f_2
[conda] ffmpeg 4.3 hf484d3e_0 pytorch
[conda] mkl 2019.4 243 anaconda
[conda] mkl-service 2.3.0 py37he904b0f_0 anaconda
[conda] mkl_fft 1.2.0 py37h23d657b_0 anaconda
[conda] mkl_random 1.0.4 py37hd81dba3_0
[conda] numpy 1.19.1 py37hbc911f0_0 anaconda
[conda] numpy-base 1.19.1 py37hfa32c7d_0 anaconda
[conda] pytorch 1.10.1 py3.7_cuda11.3_cudnn8.2.0_0 pytorch
[conda] pytorch-mutex 1.0 cuda pytorch
[conda] torchfile 0.1.0 py_0 conda-forge
[conda] torchvision 0.11.2 py37_cu113 pytorch

Start by running the minimal example with torchvision.ops.DeformConv2d and torch.nn.Conv2d using the reported versions, then compare their outputs with zero offsets. Trace the two operator entry points to identify why they diverge; done means the reproduced zero-offset case produces matching outputs.