huggingface / huggingface/lighteval

[BUG] `Registry.task_registry` not reimporting the custom task module

Open
#805 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug

If you've already imported a custom task module, then invoking `lighteval.main_tasks.list`'s call to `Registry.task_registry` will reimport the module: https://github.com/huggingface/lighteval/blob/v0.10.0/src/lighteval/tasks/registry.py#L132 --> https://github.com/huggingface/lighteval/blob/v0.10.0/src/lighteval/tasks/registry.py#L260

The issue here is then you register the metrics again in `Metrics` enum, which leads to a `TypeError` crash:

```none
TypeError: 'your_metric' already in use as property( None:
lighteval_list(custom_tasks=package.tasks.__file__)
```

This will throw:

```none
E TypeError: 'your_metric' already in use as property(, use_case=, sample_level_fn= at 0x15ad42840>, corpus_level_fn=)>)
```

## Expected behavior

LightEval to not re-import modules already imported.

## Version info

`lighteval==0.10.0`

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.