huggingface / huggingface/lighteval

[BUG] Issue with LightevalTaskConfig.stop_sequence Attribute When Unset

Open
#462 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug
When running `aimo_evals.py` from community tasks, an error occurs indicating that the Tokenizer does not accept `None` as input. This issue stems from how `stop_sequence` is handled in `LightevalTaskConfig`. Since most tasks in `lighteval` explicitly define `stop_sequence`, this problem may have gone unnoticed.

https://github.com/huggingface/lighteval/blob/0977fde60ba3564cf34c53606a53962d81eef7b0/src/lighteval/tasks/lighteval_task.py#L133

## Proposed Solution
Set the default value for stop_sequence to an empty list (`[]`) instead of `None`. Testing shows that this change resolves the issue.

``` python
self.stop_sequence = tuple(self.stop_sequence) if self.stop_sequence is not None else tuple()
```

## To Reproduce
``` sh
lighteval accelerate \
"pretrained=gpt2" \
"community|aimo_progress_prize_1|0|0" \
--custom-tasks "./community_tasks/aimo_evals.py"
```

## Expected behavior
The test should run successfully without any errors.

## Version info
This issue occurs in a version installed directly from the `main` branch using the following command:
``` sh
pip install -e ".[dev]"
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.