brain-score / brain-score/brain-score.web
duplicate scores in database
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 7
- Avg merge
- 7m
- Merged PRs (30d)
- 1
Description
this is worked around with https://github.com/brain-score/brain-score.web/pull/37 but we seem to have duplicate scores in the database `(benchmark_identifier, model)`. They are completely identical, e.g. for `mobilenet_v2_1.0_224`.
We should:
1. delete the duplicates from the database
2. prevent future duplicates from being added. I think a unique constraint `(benchmark_identifier, model)` might not work because we are adding `null` scores for erroneous runs and need a way of providing an updated score for them. We might already be doing this (https://github.com/brain-score/brain-score/blob/fcea6e19425fe256e69a5ba60580fe0e54225368/brainscore/submission/evaluation.py#L113), but we should verify
3. make sure that when there is a `null` score from an erroneous run and a newer actual score, we use the newer score for display and parent aggregation
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the workaround in brain-score.web pull request 37 and the referenced brainscore/submission/evaluation.py section around line 113. Trace how duplicate database records, null scores, display selection, and parent aggregation are handled; done means duplicates are cleaned up, future additions are prevented, and newer actual scores take precedence over erroneous null scores.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100