huggingface / huggingface/lighteval

[FT] Build in a way to specify specific IDs/Lines in Dataset to use as few-shot examples in the same split

Open
#634 3 comments 0 reactions 0 assignees View on GitHub
feature good first issue help wanted
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Solution/Feature
When creating a custom task, the few-shot samples can be specified via a split from the dataset. However, it would be convenient if there were a way to make it so I can specify samples from the split that will be used as the examples, similar to what LM Evaluation Harness does here in its yaml config:

```
test_split: train
fewshot_split: train
fewshot_config:
sampler: id_sampler
sampler_config:
id_list: ["2022_21", "2022_88", "2022_143"]
id_column: id
exclude_from_task: true

```

Something along the lines of:
```
@dataclass
class LightevalTaskConfig:
"""Stored configuration of a given [`LightevalTask`].

Arguments:
...
hf_subset (str): Subset used for the current task, will be default if none is selected.
hf_avail_splits (list[str]): All the available splits in the evaluation dataset
evaluation_splits (list[str]): List of the splits actually used for this evaluation
few_shots_split (str): Name of the split from which to sample few-shot examples
few_shots_id_column (str): Column name in the dataset to use for the few_shot_id_list
few_shots_id_list (list[str]): List of unique identifiers corresponding to the samples in the dataset, based on matching these IDs with the values in few_shots_id_column.
few_shots_select (str): Method with which to sample few-shot examples
...
"""
```

The new arguments would be `few_shots_id_column` and `few_shots_id_list`, and wrote a rough description of what they would look like.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.