brain-score / brain-score/vision

Normalize model activations before running neural similarity benchmarks

Open
#176 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
193
Forks
105
Avg merge
10h 48m
Merged PRs (30d)
10

Description

Currently the Majaj V4 and IT data is normalized to have zero mean and an std of 1 on a per channel basis. This was deemed appropriate because it is a multi-unit recording, however, to be fair the model activations should be treated the same way.

Per Martin, the cleanest place to do the normalization is probably just prior to running the neural similarity benchmarks, just after recording from the model: https://github.com/brain-score/brain-score/blob/master/brainscore/benchmarks/neural.py#L24

This is not an urgent fix, but will likely affect RDM scores to some degree.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading brainscore/benchmarks/neural.py around line 24, focusing on the point just after recording from the model and before running neural similarity benchmarks. Compare how Majaj V4 and IT data are normalized, then ensure model activations receive equivalent treatment and verify that the benchmark scores are produced successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.