brain-score / brain-score/vision
run models on ImageNet benchmark
- Dominant language
- Python
- Stars
- 193
- Forks
- 105
- Avg merge
- 10h 48m
- Merged PRs (30d)
- 10
Description
We have a crude version of the ImageNet benchmark [implemented](https://github.com/brain-score/brain-score/blob/master/brainscore/benchmarks/imagenet.py), but are currently not providing fitting stimuli (instead, we just say "imagenet", hoping the model will know how to deal with that). The proper solution would be to provide the 2012 training dataset for fitting so that the model can adapt to the task.
This would likely require us to run models on GPU and relates to https://github.com/brain-score/brain-score/issues/261: if we could launch jobs in parallel, we could mark some of them as requiring GPU.
To begin with however, we could probably get away without GPU as well though and just run models on CPU. Especially for recent submissions, we did not enter their reported ImageNet scores, so running them would be really helpful.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.