huggingface / huggingface/lighteval

[FT] Enable batched dataset_filter

Open
#322 0 comments 0 reactions 0 assignees View on GitHub
feature
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Issue encountered
When defining a custom `dataset_filter` in custom `LightevalTaskConfig` ([code](https://github.com/huggingface/lighteval/blob/7295c78fcb63604cc18c43dd62349a6120ac8600/src/lighteval/tasks/lighteval_task.py#L99)), I wanted to specify a `hf_filter` which filters the dataset by language.

It seems that by default, we do not process the examples in batches:
- entrypoint: https://github.com/huggingface/lighteval/blob/7295c78fcb63604cc18c43dd62349a6120ac8600/src/lighteval/utils/utils.py#L239
- batch is False by default [here](https://github.com/huggingface/datasets/blob/13f18e3339e8e4c8d6205e9d0a93f41b9ed39f37/src/datasets/dataset_dict.py#L1944)

More specifically, I tried to implement a vectorized filtering function, which did not work unless `batched=True`, however, it seems difficult to control this value.

My initial language filter was
```
def create_language_filter(target_language):
def language_filter(examples):
return [language == target_language for language in examples['language']]
return language_filter
```
if `dataset=dataset.filter(dataset_filter, batched=False)`, the dataset is actually not filtered by language during testing. When I ran `dataset=dataset.filter(dataset_filter, batched=True)`, the filtering was successful.

Testing code is below. Maybe this is not representative of how the lighteval task runs?
```
dataset_filter = create_language_filter(language)
dataset=dataset.filter(dataset_filter, batched=True) # switch between False and True
for i, sample in enumerate(islice(dataset, 5)):
print(f"\nSample {i + 1}:")
print(f"Language: {sample['language']}")
print(f"Text: {sample['text'][:100]}...")
```

Therefore, I modified the function as follows, but the evaluation could be slower due to single example processing during filtering?
```
def create_language_filter(target_language):
def language_filter(examples):
if isinstance(examples['language'], list):
return [language == target_language for language in examples['language']]
else:
return examples['language'] == target_language
return language_filter
```

## Solution/Feature
I am wondering if there is interest in:
1. exposing this parameter to be easily configurable
2. set batched to be default True
3. whether there is another way to run the filtering such that this isn't an issue?

Thank you!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.