huggingface / huggingface/lighteval

[BUG] sync `LightevalTaskConfig` docstring with types/defaults

Open
#748 1 comment 0 reactions 0 assignees View on GitHub
bug documentation
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug

The docstring in `LightevalTaskConfig` ([link](https://github.com/huggingface/lighteval/blob/v0.9.2/src/lighteval/tasks/lighteval_task.py#L68-L89)) lags the type hints:
- `evaluation_splits` says `list[str]` but it's actually `ListLike` so it can be `tuple`. Also, the default of `validation` isn't documented
- Imo a default of `validation` should not be there, as most people just have `train`/`test`
- `few_shots_split` says `str`, but it's actually optional with a default of `None`

There's more cases (e.g. `generation_size`) but I didn't state them here.

## To Reproduce

N/a

## Expected behavior

Up-to-date docstrings. For typed code, it's a best practice to not to restate types in the docstring for this very reason.

## Version info

```none
lighteval==0.9.2
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.