huggingface / huggingface/evaluate

Implement "text generation" task in the Evaluator

Open
#233 0 comments 0 reactions 1 assignee Claimed by @mathemakitten View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

In addition to the current task types available in the Evaluator we want a generic text generation pipeline which runs inference and returns generations. The "data" the evaluator will take in this case will be (optionally) a set of prompts for the language model. This will be useful for implementing evaluations requiring a set of model generations, such as RealToxicityPrompts's "toxicity probability" and the regard metric from [this paper](https://arxiv.org/abs/1909.01326).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.