huggingface / huggingface/lighteval

[FT] `StrEnum` for `suites` to intuitively document options

Open
#796 1 comment 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Issue encountered

As of `lighteval==0.10.0`, the `suites` list passed into `LightevalTaskConfig` is just free strings.

A free string leaves infinite options, whereas in practice only about 5 choices are used.

## Solution/Feature

A `StrEnum` be made to make `lighteval`'s code intuitive:

- Has comments/descriptions illuminating each option
- Makes it easy for devs to see all options and quickly pick the right one

```python
class Suites(StrEnum):
# Core evaluations maintained by the LightEval team
LIGHTEVAL = "lighteval"
# Third party evaluations
COMMUNITY = "community"
```

## Possible alternatives

Importable constants like the below (less preferable since it leads to the proliferation of many imports):

```python
# Core evaluations maintained by the LightEval team
LIGHTEVAL = "lighteval"
# Third party evaluations
COMMUNITY = "community"
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.