huggingface / huggingface/lighteval
[FT] Enable batched dataset_filter
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Issue encountered
When defining a custom `dataset_filter` in custom `LightevalTaskConfig` ([code](https://github.com/huggingface/lighteval/blob/7295c78fcb63604cc18c43dd62349a6120ac8600/src/lighteval/tasks/lighteval_task.py#L99)), I wanted to specify a `hf_filter` which filters the dataset by language.
It seems that by default, we do not process the examples in batches:
- entrypoint: https://github.com/huggingface/lighteval/blob/7295c78fcb63604cc18c43dd62349a6120ac8600/src/lighteval/utils/utils.py#L239
- batch is False by default [here](https://github.com/huggingface/datasets/blob/13f18e3339e8e4c8d6205e9d0a93f41b9ed39f37/src/datasets/dataset_dict.py#L1944)
More specifically, I tried to implement a vectorized filtering function, which did not work unless `batched=True`, however, it seems difficult to control this value.
My initial language filter was
```
def create_language_filter(target_language):
def language_filter(examples):
return [language == target_language for language in examples['language']]
return language_filter
```
if `dataset=dataset.filter(dataset_filter, batched=False)`, the dataset is actually not filtered by language during testing. When I ran `dataset=dataset.filter(dataset_filter, batched=True)`, the filtering was successful.
Testing code is below. Maybe this is not representative of how the lighteval task runs?
```
dataset_filter = create_language_filter(language)
dataset=dataset.filter(dataset_filter, batched=True) # switch between False and True
for i, sample in enumerate(islice(dataset, 5)):
print(f"\nSample {i + 1}:")
print(f"Language: {sample['language']}")
print(f"Text: {sample['text'][:100]}...")
```
Therefore, I modified the function as follows, but the evaluation could be slower due to single example processing during filtering?
```
def create_language_filter(target_language):
def language_filter(examples):
if isinstance(examples['language'], list):
return [language == target_language for language in examples['language']]
else:
return examples['language'] == target_language
return language_filter
```
## Solution/Feature
I am wondering if there is interest in:
1. exposing this parameter to be easily configurable
2. set batched to be default True
3. whether there is another way to run the filtering such that this isn't an issue?
Thank you!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.