NeuroTechX / NeuroTechX/moabb

compute_pvals_wilcoxon picks the one-sided tail from the mean difference, not from the signed-rank statistic

Open
#1,176 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.1k
Forks
264
Avg merge
1d 13m
Merged PRs (30d)
23

Description

moabb.analysis.meta_analysis.compute_pvals_wilcoxon turns SciPy's two-sided signed-rank p-value into a one-sided one by halving it and then choosing the tail from the sign of the mean paired difference:

p = stats.wilcoxon(df.loc[:, pipe1], df.loc[:, pipe2])[1]
p /= 2
if diffs.mean() < 0:
    p = 1 - p  # was in the other side of the distribution

The signed-rank test does not test the mean. Its direction is given by the rank sums W+ / W-, and these can disagree with the sign of the mean whenever a few large differences point one way and many small ones point the other. In that case the tail is flipped and pvals[i, j] says the opposite of what the test says.

Minimal example, 7 subjects, pipeline 1 wins on 6 of them by 0.02 and loses on one by 0.20 (SciPy 1.15.3):

import numpy as np, scipy.stats as stats

d = np.array([0.02, 0.02, 0.02, 0.02, 0.02, 0.02, -0.20])
x, y = 0.7 + d, np.full(7, 0.7)          # pipeline_1, pipeline_2

p2 = stats.wilcoxon(x, y)[1]             # 0.359375 (two-sided)
p_moabb = p2 / 2
if d.mean() < 0:
    p_moabb = 1 - p_moabb
print(p_moabb)                                            # 0.8203125  <- current moabb, "pipeline_1 > pipeline_2"
print(stats.wilcoxon(x, y, alternative="greater")[1])     # 0.1796875  <- the one-sided signed-rank test
print(stats.wilcoxon(x, y, alternative="less")[1])        # 0.9453125

compute_pvals_wilcoxon returns pvals[pipeline_1, pipeline_2] = 0.82 and pvals[pipeline_2, pipeline_1] = 0.18, i.e. it reports evidence that pipeline 2 is better, while the signed-rank test gives p = 0.18 for pipeline 1 being better and p = 0.95 for the reverse. The values then flow into compute_dataset_statistics (used for every dataset with >= perm_cutoff subjects), find_significant_differences (Stouffer combination) and summary_plot / meta_analysis_plot.

When mean and rank sums agree, the halved value still differs from the exact one-sided p by the point mass at the observed statistic (1 - p/2 vs P(W- <= w-)), which is visible in the 0.82 vs 0.95 above.

SciPy has supported alternative="greater" in scipy.stats.wilcoxon since 1.5 and moabb requires scipy>=1.9.3, so the fix is to ask SciPy for the one-sided test directly instead of reconstructing it from the two-sided value. The all-zero-differences guard (returns 0.5) and the (0, 1) clipping are unaffected. I have a patch with a regression test built on the example above and will open a PR.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at moabb.analysis.meta_analysis.compute_pvals_wilcoxon and inspect its existing SciPy Wilcoxon call and all-zero guard. Add a regression test using the seven-subject example, verify both directional p-values and clipping behavior, and run the relevant meta-analysis tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.