huggingface / huggingface/evaluate
MSE not accepting (n_samples, n_ouptuts) despite docs stating so
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
The evaluation card for [MSE](https://huggingface.co/spaces/evaluate-metric/mse) states:
---
Mandatory inputs:
- `predictions`: numeric array-like of shape (n_samples,) or (n_samples, n_outputs), representing the estimated target values.
- `references`: numeric array-like of shape (n_samples,) or (n_samples, n_outputs), representing the ground truth (correct) target values.
---
So it should be usable similar to `pytorch`'s [`MSELoss`](https://pytorch.org/docs/stable/generated/torch.nn.MSELoss.html) with multiple dimensions.
But using it with `(batch_size, multiple_outputs)` doesn't work:
```py
import evaluate
import numpy as np
m = evaluate.load('mse')
print(m.compute(predictions=np.random.rand(16,2), references=np.random.rand(16,2)))
```
⬇️
```sh
ValueError: Predictions and/or references don't match the expected format.
Expected format: {'predictions': Value(dtype='float32', id=None), 'references': Value(dtype='float32', id=None)},
Input predictions: ...
Input references: ...
```
---
Version: `evaluate==0.4.1`
Contributor guide
Assessment
This issue has not been assessed yet.