brain-score / brain-score/vision
allow users to score models without making them public?
- Dominant language
- Python
- Stars
- 193
- Forks
- 105
- Avg merge
- 10h 48m
- Merged PRs (30d)
- 10
Description
Currently, it seems that BrainScore is designed in a way that people who submit the models have to make both the model weight and network definition publicly available before getting a score (because they are on github). Would it be possible for the user to opt for just exposing the model to the BrainScore team but not making it publicly available?
This feature might be desirable for two reasons:
1. people might want to score a lot of models during the development process so that the repo will eventually contain many many redundant model entries, and everyone who uses brainscore has to download all models ever scored by all other users.
2. some users might want to make the model public only after publication, or might not want the model to be public.
Contributor guide
No contributing guide indexed for this repository
Research direction
No files or tests are named. Start by reviewing the current GitHub-based model submission and scoring flow, then determine how a model could be exposed to the BrainScore team without becoming public. Done should include a defined private-submission workflow that avoids unnecessary public model entries and downloads.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github, python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100