huggingface / huggingface/evaluate

[FR] Confidence intervals for metrics

Open
#581 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

It seems that currently simple metrics such as
```
evaluate.load(
"accuracy",
)
```
do not compute a confidence interval. This can be easily fixed by first computing the mean, and the STD, and then dividing the STD by the square of the sample count (to compute the STD of the mean estimate). (See, e.g., [here](https://blogs.sas.com/content/iml/2019/10/09/statistic-error-bars-mean.html#:~:text=the%20sample%20size%3A-,The%20SEM%20equals%20SD%20/%20sqrt(N),-.%20That%20is%2C%20the).)

Even just giving back the variance (or STD) is enough, the user can do their own computations on those.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.