huggingface / huggingface/evaluate
[FR] Confidence intervals for metrics
Open
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
It seems that currently simple metrics such as
```
evaluate.load(
"accuracy",
)
```
do not compute a confidence interval. This can be easily fixed by first computing the mean, and the STD, and then dividing the STD by the square of the sample count (to compute the STD of the mean estimate). (See, e.g., [here](https://blogs.sas.com/content/iml/2019/10/09/statistic-error-bars-mean.html#:~:text=the%20sample%20size%3A-,The%20SEM%20equals%20SD%20/%20sqrt(N),-.%20That%20is%2C%20the).)
Even just giving back the variance (or STD) is enough, the user can do their own computations on those.
Contributor guide
Assessment
This issue has not been assessed yet.