huggingface / huggingface/transformers
Easier perplexity computation
Open
Feature request
- Dominant language
- Python
- Stars
- 166k
- Forks
- 34.6k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 276
Description
# 🚀 Feature request
The docs provide a method to evaluate perplexity for a GPT-2 model, one example at a time (https://huggingface.co/transformers/perplexity.html). However this can potentially be included in the library with the computation being done in a batched manner.
## Motivation
This would make it easier and faster for people to evaluate their language models in terms of perplexity.
If not a solution integrated in the library, the example given in the docs can be updated to do computation in a batched manner for speed.
Contributor guide
Assessment
This issue has not been assessed yet.