huggingface / huggingface/evaluate

Statistically nonsensical to use skewness in `label_distribution`

Open
#659 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

The `label_distribution` measurement is supposed to quantify how biased a column of labels is towards one value or another. It computes the fraction each label value takes up out of all labels (which is useful) and, problematically, the *skewness* of that distribution:

https://github.com/huggingface/evaluate/blob/55f1bc6e072b05c2d9db1589a07e20f38902b1ec/measurements/label_distribution/label_distribution.py#L85-L93

This is statistical nonsense.

A class label is a [multinoulli variable](https://en.wikipedia.org/wiki/Categorical_distribution), which means that it's a discrete variable that can emit a finite amount of values **whose magnitude has no meaning other than being different from each other**. If you have classes `{cat, dog, giraffe}`, it has no meaning whether we choose to label `cat` as 0 or 3.1415, and it has no meaning whether `cat` is labelled 0 or `dog` is labelled 0, as long as they are different.

[*Skewness*](https://en.wikipedia.org/wiki/Skewness), however, is meant for continuous variables, and cares about magnitudes and permutations. **It looks at the symmetry of the distribution, not the uniformity.**

- The label column `[0, 0, 0, 1, 1, 1, 2, 2, 2]` has 0 skewness, because it is symmetrical.
- The label column `[0, 0, 1, 1, 1, 1, 1, 2, 2]` has 0 skewness, because it is symmetrical. Yet, clearly, there is heavy bias towards label `1`.
- The label column `[0, 0, 1, 1, 2, 2, 2, 2, 2]` is exactly as biased as the previous one (2 labels, 2 labels, 5 labels) *and yet now it has skewness != 0* because the weight of the distribution is "on the right", which has zero meaning.

The [*entropy*](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.entropy.html) of the labels is what you are looking for to measure uniformity, not skewness. Entropy is class-permutation-invariant. It is maximised for uniform distributions. If you want to normalise it, you can divide by that maximal entropy (the [Hartley function](https://en.wikipedia.org/wiki/Hartley_function)).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.