huggingface / huggingface/evaluate
Add preprocessing module for evaluation datasets
Open
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
For `boolq`, we'd need to modify each example to be in the format `f"Passage: {passage} Question: {question} Answer:` — and then appending "True" or "False" and evaluating likelihoods. Possibly via adding an f-string argument to the zero-shot evaluator.
Outstanding:
* If we do this, how should we track the formatting for reproducibility of results? Sometimes removing trailing spaces or changing capitalization makes a big difference in accuracy, but you'd need to know the exact prompting string to reproduce results
Contributor guide
Assessment
This issue has not been assessed yet.