huggingface / huggingface/evaluate
Refactor perplexity implementations to be usable with evaluators
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
Currently the `perplexity` metric and measurement both instantiate an entire model object within the `_compute()` function and run inference, which breaks the pattern where only predictions, references, and other metadata is passed in and only the metric computation is performed (e.g. inference is already run before the metric hits its `_compute` function, which is the case for most other metrics).

Evaluators already instantiate model-like objects with `model_or_pipeline`, which makes it redundant to reinstantiate _another_ copy of the model/pipeline within the `_compute()` function. It seems like this is done because perplexity can be used in a standalone manner to calculate the perplexity of some text with regards to a pretrained model (e.g. `perplexity.compute(model_id='gpt2', add_start_token=False, input_texts=input_texts)`, trading off developer flexibility for user-friendliness.
For background, this is relevant since I'm in the middle of implementing a "text-generation" evaluator, and it makes sense for perplexity to be the default metric for text generation. We _could_ alternatively write one-off logic for the text generation evaluator to compute perplexity with the model instantiated in the evaluator instead of using the metric implemented in `evaluate`, but that seems suboptimally out-of-pattern. Additionally, perplexity metrics right now are limited to be used with models which can easily instantiated via `AutoModelForCausalLM.from_pretrained(model_id)`, but does not support a generic call to a language model.
Contributor guide
Assessment
This issue has not been assessed yet.