huggingface / huggingface/evaluate
Implement "text generation" task in the Evaluator
Open
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
In addition to the current task types available in the Evaluator we want a generic text generation pipeline which runs inference and returns generations. The "data" the evaluator will take in this case will be (optionally) a set of prompts for the language model. This will be useful for implementing evaluations requiring a set of model generations, such as RealToxicityPrompts's "toxicity probability" and the regard metric from [this paper](https://arxiv.org/abs/1909.01326).
Contributor guide
Assessment
This issue has not been assessed yet.