huggingface / huggingface/lighteval
[BUG] Issue with LightevalTaskConfig.stop_sequence Attribute When Unset
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Describe the bug
When running `aimo_evals.py` from community tasks, an error occurs indicating that the Tokenizer does not accept `None` as input. This issue stems from how `stop_sequence` is handled in `LightevalTaskConfig`. Since most tasks in `lighteval` explicitly define `stop_sequence`, this problem may have gone unnoticed.
https://github.com/huggingface/lighteval/blob/0977fde60ba3564cf34c53606a53962d81eef7b0/src/lighteval/tasks/lighteval_task.py#L133
## Proposed Solution
Set the default value for stop_sequence to an empty list (`[]`) instead of `None`. Testing shows that this change resolves the issue.
``` python
self.stop_sequence = tuple(self.stop_sequence) if self.stop_sequence is not None else tuple()
```
## To Reproduce
``` sh
lighteval accelerate \
"pretrained=gpt2" \
"community|aimo_progress_prize_1|0|0" \
--custom-tasks "./community_tasks/aimo_evals.py"
```
## Expected behavior
The test should run successfully without any errors.
## Version info
This issue occurs in a version installed directly from the `main` branch using the following command:
``` sh
pip install -e ".[dev]"
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.