brain-score / brain-score/vision

alexnet_training_seed_01 appears to provide untrained (random) weights; ImageNet accuracy at chance

Open
#2,459 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
193
Forks
105
Avg merge
10h 48m
Merged PRs (30d)
10

Description

I was looking into this model since it's a leading performer on the V1 neural benchmarks. perhaps I am missing something here... but, surprisingly, it looks like the `.pth` that `alexnet_training_seed_01` loads is indistinguishable from a fresh random init of the arch: ImageNet val top-1 ~0.1% (chance). my analyses suggest that conv1 filters are basically noise, and that the features in the model look untrained as well. The state dict loads cleanly (16/16 keys, right shapes), so I didn't even catch this until I had done some analyses with this model that looked off downstream. it doesn't look like there's any corrupted file, the random weights are what the brainscore API actually provides.

the trained net exists on an [OSF repo](https://osf.io/3xupm/files/osfstorage): when converting the original TF-Slim `alexnet_v2` checkpoint myself to torch and loading their saved weights, it looks like a normal AlexNet (~54% top-1). so it appears that only `training_seed_01.pth` in the `brainscorevariability` bucket has the issue.

part of the issue is probably that the model was originally trained in tensorflow. but this is not a simple transpose/format bug during model loading: layer by layer against the TF checkpoint, per-layer weight correlation is <0.01 under the correct `HWIO -> OIHW` transpose, and the sorted value distributions don't match. they are just unrelated values, i.e. an untrained net, probably saved before the trained weights were loaded?

I only tested seed_01; the other `alexnet_training_seed_0X` models might be worth checking as well here.

Image

one question I have at this time is whether the reported results on brainscore rely on the same code for model loading as in the public-facing API, or whether the model encoding scores / ranking info may depend on other backend code that's not visible to the public? it would be very interesting if the leading model of V1 neural benchmarks in brainscore was in fact an untrained alexnet (tied with robust-resnet50?)

Image

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the ImageNet validation result with alexnet_training_seed_01 and inspecting the training_seed_01.pth artifact served by the Brain-Score API. Compare its layer weights with the trained TF-Slim alexnet_v2 checkpoint from the linked OSF repository, then check the other alexnet_training_seed_0X models. Done means the affected artifacts and the relationship between public API loading and reported scores are explained.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.