Potential Resnet Inefficiency

Hi,
I noticed a potential inefficiency with the Resnet implementation in the ordering of maxpool / ReLU operations:
https://github.com/pytorch/vision/blob/50d54a82d1479ffb6dd7469ed05fccdf290a1d84/torchvision/models/resnet.py#L189-L193

This is my thinking:
Consider a 2x2 pooling kernel. If you activate first then maxpool, you're activating 4 values then pooling for a total of 5 operations per kernel operation. However, the position of the max element in any given kernel remains the same regardless of its activation -- the value ordering relationship of elements in a kernel is maintained through activation operations). Conversely, if you maxpool first then activate, it's only 2 operations.

That is, maxpool(relu(x)) == relu(maxpool(x)) but the latter costs less computation.

Some quick testing...
Setup:

relu_first = nn.Sequential(
    nn.Conv2d(3, 512, 5),
    nn.ReLU(),
    nn.MaxPool2d(2, 2)
)

maxpool_first = nn.Sequential(
    nn.Conv2d(3, 512, 5),
    nn.MaxPool2d(2, 2),
    nn.ReLU()
)

Testing result:

Consequently, I think there would be an improvement to computation without any changes to functionality simply by reordering the forward pass operations to:

def forward(self, x):
    x = self.conv1(x)
    x = self.bn1(x)
    x = self.maxpool(x) # Changes here
    x = self.relu(x)    # Changes here

    x = self.layer1(x)
    x = self.layer2(x)
    x = self.layer3(x)
    x = self.layer4(x)

    x = self.avgpool(x)
    x = x.reshape(x.size(0), -1)
    x = self.fc(x)
    return x

Maybe there is a good reason for the choice of activation first or you haven't noticed a significant difference in real-time usage. I could be entirely missing a very important piece of thinking...! Please let me know if I am incorrect in any way.

Thanks for all your excellent work, and for your time in considering my thoughts!

Start with the linked torchvision/models/resnet.py lines 189-193 and inspect the ResNet forward ordering. Reproduce the supplied relu_first and maxpool_first benchmark, then verify that the reordered operations preserve outputs while improving computation. Done means the performance and behavior comparison support a concrete change or explain why the current order should remain.