huggingface / huggingface/evaluate

Add geometric mean of per-token-Perplexities

Open
#551 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

Hi I'm new to this and might to be overlooking smth, but it seems to me that more often than the arithmetic mean over perplexity per batch, the geometric mean over all per-token-Perplexities is used. I.e here in [huggingface's Transformers docs](https://huggingface.co/docs/transformers/perplexity). So basically just what you calculate as a single perplexity per batch but for the whole dataset. Would be helpful to have that returned here additionally.
So smth along the lines of (just to understand what i mean. Didn't test the code):
```python
ppls = []
nll_sum = 0
total_tokens = 0

for start_index in logging.tqdm(range(0, len(encoded_texts), batch_size)):

[...]

negative_log_likelihood_batch = loss_fct(shift_logits.transpose(1, 2), shift_labels) * shift_attention_mask_batch).sum(1)

perplexity_batch = torch.exp(negative_log_likelihood_batch / shift_attention_mask_batch.sum(1))

ppls += perplexity_batch.tolist()
nll_sum += negative_log_likelihood_batch.sum().item()
total_tokens += shift_attention_mask_batch.sum().item()

return {"perplexities": ppls,
"mean_perplexity": np.mean(ppls),
"geometric_mean_perplexity": torch.exp(nll_sum / total_tokens)}
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.