brain-score / brain-score/brain-score.web

view scores at specified point in time

Open
#146 0 comments 0 reactions 0 assignees View on GitHub
enhancement help wanted
Dominant language
JavaScript
Stars
3
Forks
7
Avg merge
7m
Merged PRs (30d)
1

Description

We're currently always presenting "today's" leaderboard.
But for references, it might often be useful to point at the state of the leaderboard at some point in the past.
For instance, for the 2019 CORnet paper, we might want to point to the model scores at the time. Similarly, a new paper might want to point out its state-of-the-art scores at the time of submission/publication.

Challenges:
* we do not have a timestamp of when we set benchmarks to public, so going back in time might show scores from benchmarks that were private at the time, i.e. did not show up on the leaderboard. If grouping ever changes, then we also do currently not have any way to keep track of that. I wonder if it would be better to just store the entire website each day, rather than doing this via database timestamps
* ideally we'd like to highlight certain points in time, e.g. "2020 Neuron paper", "before 2022 competition" etc. How can we design this?

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no files, tests, or entry points. Start by mapping how leaderboard scores, benchmark visibility, grouping, and timestamps are currently represented, then compare the proposed historical views with daily website snapshots. Done means a concrete, agreed design for retrieving and presenting past leaderboard states and highlighting named points in time.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
full-stack
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.