huggingface / huggingface/lighteval
[BUG] `Registry.task_registry` not reimporting the custom task module
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Describe the bug
If you've already imported a custom task module, then invoking `lighteval.main_tasks.list`'s call to `Registry.task_registry` will reimport the module: https://github.com/huggingface/lighteval/blob/v0.10.0/src/lighteval/tasks/registry.py#L132 --> https://github.com/huggingface/lighteval/blob/v0.10.0/src/lighteval/tasks/registry.py#L260
The issue here is then you register the metrics again in `Metrics` enum, which leads to a `TypeError` crash:
```none
TypeError: 'your_metric' already in use as property( None:
lighteval_list(custom_tasks=package.tasks.__file__)
```
This will throw:
```none
E TypeError: 'your_metric' already in use as property(, use_case=, sample_level_fn= at 0x15ad42840>, corpus_level_fn=)>)
```
## Expected behavior
LightEval to not re-import modules already imported.
## Version info
`lighteval==0.10.0`
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.