brain-score / brain-score/core
enforce benchmark identifier equivalent to registry identifier
- Dominant language
- Python
- Stars
- 4
- Forks
- 9
- PR merge metrics
- No merged PRs in 30d
Description
it is currently possible to construct a case like the following:
```
class MyBenchmark(BenchmarkBase):
def __init__(self):
super(MyBenchmark, self).__init__(
identifier='identifier2', ...)
benchmark_registry['identifier1'] = MyBenchmark
```
The identifier in the registry can be different from the identifier in the benchmark itself and we currently have no way of catching it.
To connect to the real world, this has lead to a bug with SyntaxGym in language for example (https://github.com/brain-score/language/pull/183).
Contributor guide
No contributing guide indexed for this repository
Research direction
No file or test is named. Search the Python package for BenchmarkBase and benchmark_registry, then trace how registration and identifiers are handled. Done means the mismatched identifier example is rejected or otherwise detected, with coverage demonstrating that behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100