huggingface / huggingface/lighteval
[FT] More general approach than `output_regex` to model answer extraction
Open
feature
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
Hi there!
Currently, we can specify an `output_regex` in the task config to extract the model answer from its generated content. This config option is used in `apply_generative_metrics`.
We could replace `output_regex` with an answer extractor object that performs arbitary operations to extract the answer. For example, that of [MMLU-Pro](https://arxiv.org/pdf/2406.01574) is as follows:
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.