huggingface / huggingface/evaluate
Perplexity for Left Padded Models
Open
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
Hi, while using the perplexity pipeline to compute the scores for `left` padded causal LMs (e.g., `gemma-2b`), I found the results are not the same if I changed the padding side to `right`. I would like to ask if we should always set `padding_side` to `right` when computing perplexity scores?
Contributor guide
Assessment
This issue has not been assessed yet.