huggingface / huggingface/transformers

Easier perplexity computation

Open
#9,648 5 comments 8 reactions 0 assignees View on GitHub
Feature request
Dominant language
Python
Stars
166k
Forks
34.6k
Avg merge
3d 8h
Merged PRs (30d)
276

Description

# 🚀 Feature request

The docs provide a method to evaluate perplexity for a GPT-2 model, one example at a time (https://huggingface.co/transformers/perplexity.html). However this can potentially be included in the library with the computation being done in a batched manner.

## Motivation

This would make it easier and faster for people to evaluate their language models in terms of perplexity.

If not a solution integrated in the library, the example given in the docs can be updated to do computation in a batched manner for speed.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.