brain-score / brain-score/brain-score.web

exclude incomplete models from comparison plot

Open
#138 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
JavaScript
Stars
3
Forks
7
Avg merge
7m
Merged PRs (30d)
1

Description

e.g. comparing IT vs behavior, a chunk of models (efficientnet-b*) were not scored on some of the IT benchmarks, leading to a score of 0 which in turn pushes down the IT average -- I think these models with missing scores should be excluded from the comparison
![image](https://user-images.githubusercontent.com/5308236/161111513-a9796039-c474-4d8e-8068-9f562d29e4f3.png)

related to #136

Contributor guide

No contributing guide indexed for this repository

Research direction

No file, test, or entry point is named. Start by locating the comparison plot and the code that turns missing benchmark scores into zeroes; inspect how IT and behavior averages are computed. Done means models missing scores for some benchmarks no longer lower the comparison averages, with the incomplete-model case covered by the relevant check.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
data-visualization
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.