huggingface / huggingface/evaluate

Missing model caching in perplexity metrics

Open
#402 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

The "official" implementation of `perplexity` metric does not cache language model [[1][1]]. It seems that metric instance should fetch model and prepare it for further usage in `_download_and_prepare`. I suppose that there should be clear API on caching and cache resetting. Also, it is totally unclear how to configure metrics on loading (there is only `config_name` but `kwargs` are just ignored).

[1]: https://github.com/huggingface/evaluate/blob/6abb0d53b82b1e5efea5d683b91d7990a653c78d/metrics/perplexity/perplexity.py#L103-L115

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.