brain-score / brain-score/vision

ActivationsExtractorHelper._package_layer infers wrong shape for 1x1 conv implemented as linear layers

Open
#2,190 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
193
Forks
105
Avg merge
10h 48m
Merged PRs (30d)
10

Description

ConvNeXt (and possibly other modern models) implement 1x1 convolutions as linear layers broadcasted over height and width because this is slightly more performant (according to a comment in the model repo).
E.g. from the [ConvNeXt implementation in the FacebookResearch Repo](https://github.com/facebookresearch/ConvNeXt/blob/main/models/convnext.py):
Line 30: `self.pwconv1 = nn.Linear(dim, 4 * dim) # pointwise/1x1 convs, implemented with linear layers`
In order to achieve this, dimensions are temporarily permuted:
Line 40: `x = x.permute(0, 2, 3, 1) # (N, C, H, W) -> (N, H, W, C)`

This leads to activations that match with the Conv2D case in the `_package_layer` function [line 281-282](https://github.com/brain-score/vision/blob/aa6797eac70b5dda70ef35653bdc9b5f5a56a089/brainscore_vision/model_helpers/activations/core.py#L281) but do not have channels first:
```
elif flatten_indices.shape[1] == 3: # 2DConv, e.g. resnet
flatten_coord_names = ['channel', 'channel_x', 'channel_y']
```
Thus for models that have committed a 1x1 conv layer for any brain region, the wrong shape is inferred.
E.g. for ConvNeXt_xlarge ([from the timm implementation, but same principle](https://github.com/huggingface/pytorch-image-models/blob/main/timm/models/convnext.py)) V1-region, it infers
channels: 14
x-dim: 14
y-dim: 4096

Image

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.