brain-score / brain-score/vision

allow users to score models without making them public?

Open
#610 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
193
Forks
105
Avg merge
10h 48m
Merged PRs (30d)
10

Description

Currently, it seems that BrainScore is designed in a way that people who submit the models have to make both the model weight and network definition publicly available before getting a score (because they are on github). Would it be possible for the user to opt for just exposing the model to the BrainScore team but not making it publicly available?

This feature might be desirable for two reasons:
1. people might want to score a lot of models during the development process so that the repo will eventually contain many many redundant model entries, and everyone who uses brainscore has to download all models ever scored by all other users.
2. some users might want to make the model public only after publication, or might not want the model to be public.

Contributor guide

No contributing guide indexed for this repository

Research direction

No files or tests are named. Start by reviewing the current GitHub-based model submission and scoring flow, then determine how a model could be exposed to the BrainScore team without becoming public. Done should include a defined private-submission workflow that avoids unnecessary public model entries and downloads.

Written by the indexing model from the issue text.

Assessment

Tech stack
github, python
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.