brain-score / brain-score/core

enforce benchmark identifier equivalent to registry identifier

Open
#35 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
4
Forks
9
PR merge metrics
No merged PRs in 30d

Description

it is currently possible to construct a case like the following:
```
class MyBenchmark(BenchmarkBase):
def __init__(self):
super(MyBenchmark, self).__init__(
identifier='identifier2', ...)

benchmark_registry['identifier1'] = MyBenchmark
```
The identifier in the registry can be different from the identifier in the benchmark itself and we currently have no way of catching it.

To connect to the real world, this has lead to a bug with SyntaxGym in language for example (https://github.com/brain-score/language/pull/183).

Contributor guide

No contributing guide indexed for this repository

Research direction

No file or test is named. Search the Python package for BenchmarkBase and benchmark_registry, then trace how registration and identifiers are handled. Done means the mismatched identifier example is rejected or otherwise detected, with coverage demonstrating that behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.