brain-score / brain-score/vision
Interface to easily programmatically score many models locally on public benchmarks
- Dominant language
- Python
- Stars
- 193
- Forks
- 105
- Avg merge
- 10h 48m
- Merged PRs (30d)
- 10
Description
After the update, it seems that brain-score deprecated the interface to easily programmatically score many models locally on public benchmarks. This is essentially the function achieved by this [tutorial](https://github.com/brain-score/candidate_models/blob/master/examples/score-model.ipynb) but now it is deprecated.
As a user of brain score, I think this use case is still very desirable because it is often the case that users want to do some quick prototyping of many different models (could be the same model architecture but trained with different hyperparameters, etc.) on a small number of public benchmarks before submitting to the brain-score full benchmark. When doing quick prototyping, users may want to keep the code for model definition (defining pytorch models, loading weights, etc.) in their own repository instead of registering the model in the brainscore_vision/models (writing all the init.py files, etc.) each time they want to score a new model with perhaps just a small variation, because this could be tedious and time-consuming. It would be great if brain score could still support this [use case](https://github.com/brain-score/candidate_models/blob/master/examples/score-model.ipynb) or perhaps use some script to automate model registration process (writing all the init.py files in brainscore_vision/models, etc.) while allowing users to keep the model definition in their own repo.
I figured out a little hack to make it work though, but this may not be ideal since it bypasses many new features such as dependency check and score saving. (https://github.com/brain-score/vision/issues/533#issuecomment-1937820878)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the deprecated workflow described in examples/score-model.ipynb and compare it with the hack in issue #533, including the references to brainscore_vision/models. Define the supported local-scoring path for externally defined models and verify that dependency checks and score saving are not bypassed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- developer-experience, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100