huggingface / huggingface/lighteval
How to calculate perplexity from an OpenAI compatible API
Open
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
Hello,
I'm new to LightEval. I want to use LightEval to evaluate an LLM model that is served via an API. The API is OpenAI compatible. It also returns logprobs for each token. Is there a built-in function to evaluate the perplexity score? I'm asking because I see that it’s not implemented.
https://github.com/huggingface/lighteval/blob/d805f9fa0a84da9ca4c0c6a638bbed149a7012a3/src/lighteval/models/litellm_model.py#L322
Any help or guidance is greatly appreciated. Thanks.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.