huggingface / huggingface/lighteval

[BUG] `test` split forced to hit `Careful` warning

Open
#801 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug

Using `hf_avail_splits=["test"]` with `LightevalTaskConfig` you are forced to hit:

```none
[2025-06-06 18:21:10,501] [ WARNING]: Careful, the task community|foo is using evaluation data to build the few shot examples. (lighteval_task.py:260)
```

## To Reproduce

Since `"test"` isn't a value in `allowed_splits` at https://github.com/huggingface/lighteval/blob/v0.10.0/src/lighteval/tasks/lighteval_task.py#L251.

## Expected behavior

Not forced to hit a warning with `hf_avail_splits=["test"]`, since `test` is a commonly used split.

## Version info

`lighteval==0.10.0`

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.