brain-score / brain-score/vision
Normalize model activations before running neural similarity benchmarks
- Dominant language
- Python
- Stars
- 193
- Forks
- 105
- Avg merge
- 10h 48m
- Merged PRs (30d)
- 10
Description
Currently the Majaj V4 and IT data is normalized to have zero mean and an std of 1 on a per channel basis. This was deemed appropriate because it is a multi-unit recording, however, to be fair the model activations should be treated the same way.
Per Martin, the cleanest place to do the normalization is probably just prior to running the neural similarity benchmarks, just after recording from the model: https://github.com/brain-score/brain-score/blob/master/brainscore/benchmarks/neural.py#L24
This is not an urgent fix, but will likely affect RDM scores to some degree.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading brainscore/benchmarks/neural.py around line 24, focusing on the point just after recording from the model and before running neural similarity benchmarks. Compare how Majaj V4 and IT data are normalized, then ensure model activations receive equivalent treatment and verify that the benchmark scores are produced successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100