brain-score / brain-score/vision
alexnet_training_seed_01 appears to provide untrained (random) weights; ImageNet accuracy at chance
- Dominant language
- Python
- Stars
- 193
- Forks
- 105
- Avg merge
- 10h 48m
- Merged PRs (30d)
- 10
Description
I was looking into this model since it's a leading performer on the V1 neural benchmarks. perhaps I am missing something here... but, surprisingly, it looks like the `.pth` that `alexnet_training_seed_01` loads is indistinguishable from a fresh random init of the arch: ImageNet val top-1 ~0.1% (chance). my analyses suggest that conv1 filters are basically noise, and that the features in the model look untrained as well. The state dict loads cleanly (16/16 keys, right shapes), so I didn't even catch this until I had done some analyses with this model that looked off downstream. it doesn't look like there's any corrupted file, the random weights are what the brainscore API actually provides.
the trained net exists on an [OSF repo](https://osf.io/3xupm/files/osfstorage): when converting the original TF-Slim `alexnet_v2` checkpoint myself to torch and loading their saved weights, it looks like a normal AlexNet (~54% top-1). so it appears that only `training_seed_01.pth` in the `brainscorevariability` bucket has the issue.
part of the issue is probably that the model was originally trained in tensorflow. but this is not a simple transpose/format bug during model loading: layer by layer against the TF checkpoint, per-layer weight correlation is <0.01 under the correct `HWIO -> OIHW` transpose, and the sorted value distributions don't match. they are just unrelated values, i.e. an untrained net, probably saved before the trained weights were loaded?
I only tested seed_01; the other `alexnet_training_seed_0X` models might be worth checking as well here.
one question I have at this time is whether the reported results on brainscore rely on the same code for model loading as in the public-facing API, or whether the model encoding scores / ranking info may depend on other backend code that's not visible to the public? it would be very interesting if the leading model of V1 neural benchmarks in brainscore was in fact an untrained alexnet (tied with robust-resnet50?)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the ImageNet validation result with alexnet_training_seed_01 and inspecting the training_seed_01.pth artifact served by the Brain-Score API. Compare its layer weights with the trained TF-Slim alexnet_v2 checkpoint from the linked OSF repository, then check the other alexnet_training_seed_0X models. Done means the affected artifacts and the relationship between public API loading and reported scores are explained.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100