compute_pvals_wilcoxon picks the one-sided tail from the mean difference, not from the signed-rank statistic
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 264
- Avg merge
- 1d 13m
- Merged PRs (30d)
- 23
Description
moabb.analysis.meta_analysis.compute_pvals_wilcoxon turns SciPy's two-sided signed-rank p-value into a one-sided one by halving it and then choosing the tail from the sign of the mean paired difference:
p = stats.wilcoxon(df.loc[:, pipe1], df.loc[:, pipe2])[1]
p /= 2
if diffs.mean() < 0:
p = 1 - p # was in the other side of the distribution
The signed-rank test does not test the mean. Its direction is given by the rank sums W+ / W-, and these can disagree with the sign of the mean whenever a few large differences point one way and many small ones point the other. In that case the tail is flipped and pvals[i, j] says the opposite of what the test says.
Minimal example, 7 subjects, pipeline 1 wins on 6 of them by 0.02 and loses on one by 0.20 (SciPy 1.15.3):
import numpy as np, scipy.stats as stats
d = np.array([0.02, 0.02, 0.02, 0.02, 0.02, 0.02, -0.20])
x, y = 0.7 + d, np.full(7, 0.7) # pipeline_1, pipeline_2
p2 = stats.wilcoxon(x, y)[1] # 0.359375 (two-sided)
p_moabb = p2 / 2
if d.mean() < 0:
p_moabb = 1 - p_moabb
print(p_moabb) # 0.8203125 <- current moabb, "pipeline_1 > pipeline_2"
print(stats.wilcoxon(x, y, alternative="greater")[1]) # 0.1796875 <- the one-sided signed-rank test
print(stats.wilcoxon(x, y, alternative="less")[1]) # 0.9453125
compute_pvals_wilcoxon returns pvals[pipeline_1, pipeline_2] = 0.82 and pvals[pipeline_2, pipeline_1] = 0.18, i.e. it reports evidence that pipeline 2 is better, while the signed-rank test gives p = 0.18 for pipeline 1 being better and p = 0.95 for the reverse. The values then flow into compute_dataset_statistics (used for every dataset with >= perm_cutoff subjects), find_significant_differences (Stouffer combination) and summary_plot / meta_analysis_plot.
When mean and rank sums agree, the halved value still differs from the exact one-sided p by the point mass at the observed statistic (1 - p/2 vs P(W- <= w-)), which is visible in the 0.82 vs 0.95 above.
SciPy has supported alternative="greater" in scipy.stats.wilcoxon since 1.5 and moabb requires scipy>=1.9.3, so the fix is to ask SciPy for the one-sided test directly instead of reconstructing it from the two-sided value. The all-zero-differences guard (returns 0.5) and the (0, 1) clipping are unaffected. I have a patch with a regression test built on the example above and will open a PR.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at moabb.analysis.meta_analysis.compute_pvals_wilcoxon and inspect its existing SciPy Wilcoxon call and all-zero guard. Add a regression test using the seven-subject example, verify both directional p-values and clipping behavior, and run the relevant meta-analysis tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100