brain-score / brain-score/brain-score.web

duplicate scores in database

Open
#38 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
3
Forks
7
Avg merge
7m
Merged PRs (30d)
1

Description

this is worked around with https://github.com/brain-score/brain-score.web/pull/37 but we seem to have duplicate scores in the database `(benchmark_identifier, model)`. They are completely identical, e.g. for `mobilenet_v2_1.0_224`.

We should:
1. delete the duplicates from the database
2. prevent future duplicates from being added. I think a unique constraint `(benchmark_identifier, model)` might not work because we are adding `null` scores for erroneous runs and need a way of providing an updated score for them. We might already be doing this (https://github.com/brain-score/brain-score/blob/fcea6e19425fe256e69a5ba60580fe0e54225368/brainscore/submission/evaluation.py#L113), but we should verify
3. make sure that when there is a `null` score from an erroneous run and a newer actual score, we use the newer score for display and parent aggregation

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the workaround in brain-score.web pull request 37 and the referenced brainscore/submission/evaluation.py section around line 113. Trace how duplicate database records, null scores, display selection, and parent aggregation are handled; done means duplicates are cleaned up, future additions are prevented, and newer actual scores take precedence over erroneous null scores.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.