statsmodels / statsmodels/statsmodels

ENH: pairwise comparison, more general than tukey-hsd

Open
#9,478 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

comp-stats reference type-enh
Dominant language
Python
Stars
11.6k
Forks
3.6k
Avg merge
7h 37m
Merged PRs (30d)
96

Description

follow-up to #9175 adding games-howell
related issue #9472 distribution needed for Dunnet's T3

Sauder and DeMars 2019 looks good for a more recent Monte Carlo comparison of multiple tests handling in oneway pairwise comparison
e..g. tukey-hsd looks good as long as variances are equal (group sizes can vary)
many older methods are bad, don't have approximately correct power.

methods robust to unequal group sizes and unequal variances
games-howell
Dunnet's C and T3
Tamhance's T2
in most cases they are conservative for the cases simulated in the Monte Carlo (e.g. group sizes are not very small)

Games-Howell is doing well enough to be added as main method in statsmodels, and is recommended in some references as long as group sizes are not very small.

All others are very conservative if the largest group has the largest variance, and
are very liberal if the smallest group has the largest variance.
Some methods are ok if either group sizes are the same and variances differ, or reversed, group sizes differ and variances are the same.

Sauder, Derek C., and Christine E. DeMars. 2019. “An Updated Recommendation for Multiple Comparisons.” Advances in Methods and Practices in Psychological Science 2 (1): 26–44. https://doi.org/10.1177/2515245918808784.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no files or tests. Start by reading #9175 and #9472 and the cited Sauder and DeMars 2019 paper; clarify the intended API and scope for a general pairwise comparison before implementation. Done should include an agreed method and required distribution support rather than only the existing Tukey-HSD case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.