huggingface / huggingface/evaluate
Missing model caching in perplexity metrics
Open
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
The "official" implementation of `perplexity` metric does not cache language model [[1][1]]. It seems that metric instance should fetch model and prepare it for further usage in `_download_and_prepare`. I suppose that there should be clear API on caching and cache resetting. Also, it is totally unclear how to configure metrics on loading (there is only `config_name` but `kwargs` are just ignored).
[1]: https://github.com/huggingface/evaluate/blob/6abb0d53b82b1e5efea5d683b91d7990a653c78d/metrics/perplexity/perplexity.py#L103-L115
Contributor guide
Assessment
This issue has not been assessed yet.