huggingface / huggingface/lighteval
[FT] `StrEnum` for `suites` to intuitively document options
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Issue encountered
As of `lighteval==0.10.0`, the `suites` list passed into `LightevalTaskConfig` is just free strings.
A free string leaves infinite options, whereas in practice only about 5 choices are used.
## Solution/Feature
A `StrEnum` be made to make `lighteval`'s code intuitive:
- Has comments/descriptions illuminating each option
- Makes it easy for devs to see all options and quickly pick the right one
```python
class Suites(StrEnum):
# Core evaluations maintained by the LightEval team
LIGHTEVAL = "lighteval"
# Third party evaluations
COMMUNITY = "community"
```
## Possible alternatives
Importable constants like the below (less preferable since it leads to the proliferation of many imports):
```python
# Core evaluations maintained by the LightEval team
LIGHTEVAL = "lighteval"
# Third party evaluations
COMMUNITY = "community"
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.