huggingface / huggingface/evaluate

Allow for specify coda device in perplexity evaluation

Open
#568 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

The current implementation of `perplexity.compute` does not allow to provide a specific coda device (e.g. `cuda:1` instead of just `cuda`.

https://github.com/huggingface/evaluate/blob/8dfe05784099fb9af55b8e77793205a3b7c86465/metrics/perplexity/perplexity.py#L108

Should probably be changed to something along the lines of:
```python
assert device in ["gpu", "cpu", "cuda"] or re.search(r'^cuda:[0-9]+$', device), "device should be either gpu or cpu."
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.