brain-score / brain-score/vision

run models on ImageNet benchmark

Open
#262 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
193
Forks
105
Avg merge
10h 48m
Merged PRs (30d)
10

Description

We have a crude version of the ImageNet benchmark [implemented](https://github.com/brain-score/brain-score/blob/master/brainscore/benchmarks/imagenet.py), but are currently not providing fitting stimuli (instead, we just say "imagenet", hoping the model will know how to deal with that). The proper solution would be to provide the 2012 training dataset for fitting so that the model can adapt to the task.

This would likely require us to run models on GPU and relates to https://github.com/brain-score/brain-score/issues/261: if we could launch jobs in parallel, we could mark some of them as requiring GPU.
To begin with however, we could probably get away without GPU as well though and just run models on CPU. Especially for recent submissions, we did not enter their reported ImageNet scores, so running them would be really helpful.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.