SUMM/ENH: pairwise comparisons, methods, multiplicity correction
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 20/100
Research direction
Start by reviewing the existing tukeyhsd function, the MultiConparison class, and the oneway variance option, along with the related #7332 pull request and #8168 issue. Compare the supported pairwise methods and variance assumptions with the cited SPSS and Sauder–DeMars references. Done would require an agreed generic pairwise-testing design, validated p-values or multiplicity correction, and tests for equal- and unequal-variance cases.
Written by the indexing model from the issue text.
Description
#7332 PR for games howell
#8168 issue for snk, dunnet-t
currently we only have tukey-hsd as supported method
we have generic allpairs test but without specific p-values like studentized range, requires use of pvalue mutliplicity correction
This article has an overview with formulas in the online supplement
formulas are in and from spss manual IBM_SPSS_Statistics_Algorithms_28.0.0.pdf appendix G
Sauder, Derek C., and Christine E. DeMars. 2019. “An Updated Recommendation for Multiple Comparisons.” Advances in Methods and Practices in Psychological Science 2 (1): 26–44. https://doi.org/10.1177/2515245918808784.
tukey-hsd uses pooled variance (variance of residual from group-demeaning), i.e. assumes homoscedasticity
games-howell and others allow for heteroscedasticity
Sauder, DeMars list 18 methods available in SPSS, 4 of which are for heteroscedasticity
The second distinction is in sequential versus simultaneous methods.
The tukeyhsd function in the sandbox already a switch whether variances are assumed to be equal. However, the method in the MultiConparison class only uses the pooled variance.
Aside: I'm not sure whether the path for unequal variance in the tukeyhsd function is correct. I remember having some bugs in parts that were not used yet at the time.
So I guess we can make the pairwise testing function more generic and use keyword option for the variance assumptions similar to what I did in oneway. We can keep alias functions for tukey-hsd and games-howell for name recognition.
Optionally, we can use pvalue multiple testing correction instead of using studentized range distribution.
(I will have to work my way through the details of those methods.)
supporting code
#7332 also has plot and letter functions, I have letter assingment in another PR
pairwise comparison do not produce a partition of groups.
(I still have some old code in the sandbox, where I looked at getting partitions)
other articles that I just saw use AIC or model selection criteria to estimate models that partition the set of groups.
might be worth a look
If we use OLS (or other models) and information criteria to select a model, then all we need to to is to create the design matrix for each partition, (or sequence of partitions?)
Cribbie, Robert A., and H. J. Keselman. 2003. “Pairwise Multiple Comparisons: A Model Comparison Approach versus Stepwise Procedures.” British Journal of Mathematical and Statistical Psychology 56 (1): 167–82. https://doi.org/10.1348/000711003321645412.
Dayton, C. Mitchell. 1998. “Information Criteria for the Paired-Comparisons Problem.” The American Statistician 52 (2): 144–51. https://doi.org/10.1080/00031305.1998.10480554.
- Dominant language
- Python
- Stars
- 11.6k
- Forks
- 3.6k
- Avg merge
- 7h 37m
- Merged PRs (30d)
- 96
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from statsmodels/statsmodels
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
statsmodels/statsmodels#10271 ·
-
type-bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
statsmodels/statsmodels#10269 ·
-
Documentation
Difficulty 2/5 1-3 hours Newbie friendliness 92/100
statsmodels/statsmodels#10266 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 78/100
statsmodels/statsmodels#9627 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
statsmodels/statsmodels#9293 · 1 comment ·
All issues in statsmodels/statsmodels
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
bancolombia/sentinel#23 ·
-
test md OpenCI
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
integration:quickjs org:external priority:backlog topic:code-interpreter topic:middleware type:feature
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
langchain-ai/deepagents#6450 ·
-
bug client
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100