huggingface / huggingface/evaluate

Statistical significance testing

Open
#16 1 comment 0 reactions 0 assignees View on GitHub
metric request
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

As mentioned in a recent [paper](https://aclanthology.org/2021.acl-long.566.pdf) about evaluating MT approaches (and probably other sources too), statistical significance testing can be used to confirm that one method is superior to another, saying that 'it remains one of the most cost-effective tools to check how trustworthy a particular difference between two metric scores is."

We could possibly use the Wilcoxon signed-rank test (implemented in [scipy](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.wilcoxon.html)) or another similar approach.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.