SUMM/ENH: pairwise comparisons, methods, multiplicity correction

Open
#8,198 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
20/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale
Tech stack
python
Domain
data

Research direction

Start by reviewing the existing tukeyhsd function, the MultiConparison class, and the oneway variance option, along with the related #7332 pull request and #8168 issue. Compare the supported pairwise methods and variance assumptions with the cited SPSS and Sauder–DeMars references. Done would require an agreed generic pairwise-testing design, validated p-values or multiplicity correction, and tests for equal- and unequal-variance cases.

Written by the indexing model from the issue text.

Description

comp-stats type-enh

#7332 PR for games howell
#8168 issue for snk, dunnet-t

currently we only have tukey-hsd as supported method
we have generic allpairs test but without specific p-values like studentized range, requires use of pvalue mutliplicity correction

This article has an overview with formulas in the online supplement
formulas are in and from spss manual IBM_SPSS_Statistics_Algorithms_28.0.0.pdf appendix G

Sauder, Derek C., and Christine E. DeMars. 2019. “An Updated Recommendation for Multiple Comparisons.” Advances in Methods and Practices in Psychological Science 2 (1): 26–44. https://doi.org/10.1177/2515245918808784.

tukey-hsd uses pooled variance (variance of residual from group-demeaning), i.e. assumes homoscedasticity
games-howell and others allow for heteroscedasticity

Sauder, DeMars list 18 methods available in SPSS, 4 of which are for heteroscedasticity

The second distinction is in sequential versus simultaneous methods.

The tukeyhsd function in the sandbox already a switch whether variances are assumed to be equal. However, the method in the MultiConparison class only uses the pooled variance.
Aside: I'm not sure whether the path for unequal variance in the tukeyhsd function is correct. I remember having some bugs in parts that were not used yet at the time.

So I guess we can make the pairwise testing function more generic and use keyword option for the variance assumptions similar to what I did in oneway. We can keep alias functions for tukey-hsd and games-howell for name recognition.
Optionally, we can use pvalue multiple testing correction instead of using studentized range distribution.

(I will have to work my way through the details of those methods.)


supporting code
#7332 also has plot and letter functions, I have letter assingment in another PR


pairwise comparison do not produce a partition of groups.
(I still have some old code in the sandbox, where I looked at getting partitions)

other articles that I just saw use AIC or model selection criteria to estimate models that partition the set of groups.
might be worth a look
If we use OLS (or other models) and information criteria to select a model, then all we need to to is to create the design matrix for each partition, (or sequence of partitions?)

Cribbie, Robert A., and H. J. Keselman. 2003. “Pairwise Multiple Comparisons: A Model Comparison Approach versus Stepwise Procedures.” British Journal of Mathematical and Statistical Psychology 56 (1): 167–82. https://doi.org/10.1348/000711003321645412.

Dayton, C. Mitchell. 1998. “Information Criteria for the Paired-Comparisons Problem.” The American Statistician 52 (2): 144–51. https://doi.org/10.1080/00031305.1998.10480554.

Dominant language
Python
Stars
11.6k
Forks
3.6k
Avg merge
7h 37m
Merged PRs (30d)
96

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from statsmodels/statsmodels

All issues in statsmodels/statsmodels

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.