brain-score / brain-score/brain-score.web
automatically run new benchmarks / missing scores
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 7
- Avg merge
- 7m
- Merged PRs (30d)
- 1
Description
when new benchmarks are added, we currently have to manually re-run models on those benchmarks. It would be nice if we could either automatically rerun models as new benchmarks are added (sounds more difficult) or have a button in the profile that lets us automatically run all missing scores
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the existing model benchmark execution flow and the profile UI, since the issue names no files or tests. Compare the two proposed approaches and define how missing scores are detected, how reruns are triggered, and what completion should look like before implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- full-stack
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100