brain-score / brain-score/vision

Interface to easily programmatically score many models locally on public benchmarks

Open
#611 9 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
193
Forks
105
Avg merge
10h 48m
Merged PRs (30d)
10

Description

After the update, it seems that brain-score deprecated the interface to easily programmatically score many models locally on public benchmarks. This is essentially the function achieved by this [tutorial](https://github.com/brain-score/candidate_models/blob/master/examples/score-model.ipynb) but now it is deprecated.

As a user of brain score, I think this use case is still very desirable because it is often the case that users want to do some quick prototyping of many different models (could be the same model architecture but trained with different hyperparameters, etc.) on a small number of public benchmarks before submitting to the brain-score full benchmark. When doing quick prototyping, users may want to keep the code for model definition (defining pytorch models, loading weights, etc.) in their own repository instead of registering the model in the brainscore_vision/models (writing all the init.py files, etc.) each time they want to score a new model with perhaps just a small variation, because this could be tedious and time-consuming. It would be great if brain score could still support this [use case](https://github.com/brain-score/candidate_models/blob/master/examples/score-model.ipynb) or perhaps use some script to automate model registration process (writing all the init.py files in brainscore_vision/models, etc.) while allowing users to keep the model definition in their own repo.

I figured out a little hack to make it work though, but this may not be ideal since it bypasses many new features such as dependency check and score saving. (https://github.com/brain-score/vision/issues/533#issuecomment-1937820878)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the deprecated workflow described in examples/score-model.ipynb and compare it with the hack in issue #533, including the references to brainscore_vision/models. Define the supported local-scoring path for externally defined models and verify that dependency checks and score saving are not bypassed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
developer-experience, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.