huggingface / huggingface/evaluate

MSE not accepting (n_samples, n_ouptuts) despite docs stating so

Open
#550 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

The evaluation card for [MSE](https://huggingface.co/spaces/evaluate-metric/mse) states:

---

Mandatory inputs:
- `predictions`: numeric array-like of shape (n_samples,) or (n_samples, n_outputs), representing the estimated target values.
- `references`: numeric array-like of shape (n_samples,) or (n_samples, n_outputs), representing the ground truth (correct) target values.

---

So it should be usable similar to `pytorch`'s [`MSELoss`](https://pytorch.org/docs/stable/generated/torch.nn.MSELoss.html) with multiple dimensions.
But using it with `(batch_size, multiple_outputs)` doesn't work:

```py
import evaluate
import numpy as np

m = evaluate.load('mse')
print(m.compute(predictions=np.random.rand(16,2), references=np.random.rand(16,2)))
```
⬇️
```sh
ValueError: Predictions and/or references don't match the expected format.
Expected format: {'predictions': Value(dtype='float32', id=None), 'references': Value(dtype='float32', id=None)},
Input predictions: ...
Input references: ...
```

---

Version: `evaluate==0.4.1`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.