Maybe replace scipy.stats.mannwhitneyu?
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 302
- PR merge metrics
- No merged PRs in 30d
Description
I don't know if this issue was considered when implementing the analysis module, but using scipy.stats to compute Mann-Whitney U is not ideal.
I first noticed the warning on the [documentation page](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.mannwhitneyu.html), not to use with sample sizes less than 20. I then found the [open pull request](https://github.com/scipy/scipy/pull/4933) that seems to be completely abandoned and a [warning](https://github.com/scipy/scipy/issues/11035#issuecomment-552508317) from one of the maintainers that "_the current implementation is pretty badly broken and needs a complete rewrite I'm afraid_".
There's a lot of controversy about using null-hypothesis significance tests in general, but I'm not attempting to bring in that debate. I just wanted to suggest that if Fuzzbench is using Mann-Whitney U, then it should use an implementation that computes p-values with the exact distribution since the sample sizes are generally <= 20.
Unfortunately I don't have a good suggestion as a replacement. I thought the `wilcox.test` in R would be good replacement, but even that implementation only uses the exact method when the sample sizes are less than 50 and there are **no** duplicates anywhere in the data. I would expect most data sets to have duplicates, either from one fuzzer getting the same coverage in two trials or from different fuzzers achieving the exact same coverage.
The Vargha Delaney A measure has been used and recommended in fuzzing research, but doesn't really replace Mann-Whitney as a statistical test concerning the distributions.
Does this matter for Fuzzbench data? I did some quick analysis on 2020-11-01 experiment just to check Scipy versus R.
I found a couple changes in the significance level around the `0.01` level, but I did not find changes at the `0.05` level.
Here's one example below. With scipy.stats, the significance level is 0.05 since the p-value is greater than 0.01, but with the exact method in R, the p-value is 0.009. There's certainly not a large delta between the values, but changes the significance level due to the cutoff values. In this instance, there are no duplicate values, so R is using the exact method to determine the p-value.
- first compute using Scipy.stats
```python
>>> df = exp_snapshot_df
>>> app_fast_v2 = sorted(df[(df.fuzzer =='aflplusplus_fast_v2') & (df.benchmark == 'freetype2-2017')].edges_covered)
>>> app = sorted(df[(df.fuzzer =='aflplusplus') & (df.benchmark == 'freetype2-2017')].edges_covered)
>>> app_fast_v2
[25226, 25648, 25783, 25936, 26108, 26227, 26261, 26325, 26425, 26434, 26563, 26598, 26644, 26689, 26715, 26932, 27075, 27089, 27173, 27511]
>>> app
[24970, 25301, 25323, 25563, 25578, 25617, 25721, 25779, 25824, 25856, 25910, 26052, 26210, 26304, 26310, 26463, 26590, 26629, 26840, 26950]
>>> import scipy.stats as ss
>>> ss.mannwhitneyu(app_fast_v2, app, alternative='two-sided')
MannwhitneyuResult(statistic=295.0, pvalue=0.010581211443165648)
```
- Now compute MW in R.
```R
> app_fast_v2 = c(25226, 25648, 25783, 25936, 26108, 26227, 26261, 26325, 26425, 26434, 26563, 26598, 26644, 26689, 26715, 26932, 27075, 27089, 27173, 27511)
> app = c(24970, 25301, 25323, 25563, 25578, 25617, 25721, 25779, 25824, 25856, 25910, 26052, 26210, 26304, 26310, 26463, 26590, 26629, 26840, 26950)
> wilcox.test(app_fast_v2, app, alternative="two.sided", exact=TRUE)
Wilcoxon rank sum test
data: app and app_fast_v2
W = 105, p-value = 0.009484
alternative hypothesis: true location shift is not equal to 0
```
Contributor guide
Assessment
This issue has not been assessed yet.