easystats / easystats/parameters

Inconsistency in p_adjust when working with emmeans/emGrid objects

Open
#854 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Enhancement :boom:
Dominant language
R
Stars
499
Forks
45
Avg merge
3d 1h
Merged PRs (30d)
3

Description

Let's assume I have an emmeans/emGrid object already created:

> emg
 Week    emmean    SE  df asymp.LCL asymp.UCL
 Week 1  -0.317 0.132 Inf    -0.577   -0.0573
 Week 2  -0.633 0.136 Inf    -0.900   -0.3656
 Week 4  -0.719 0.178 Inf    -1.068   -0.3696
 Week 8  -0.984 0.166 Inf    -1.309   -0.6587
 Week 16 -1.198 0.162 Inf    -1.514   -0.8810
 Week 56 -1.456 0.290 Inf    -2.025   -0.8879

Covariance estimate used: vbeta 
Confidence level used: 0.95 

Now I want both CI + p-value to be adjusted simultaneously using the mutivariate t distribution:

> set.seed(1000); update(emg, infer = c(TRUE, TRUE), adjust="mvt")
 Week    emmean    SE  df asymp.LCL asymp.UCL z.ratio p.value
 Week 1  -0.317 0.132 Inf    -0.652    0.0181  -2.393  0.0717
 Week 2  -0.633 0.136 Inf    -0.977   -0.2880  -4.643  <.0001
 Week 4  -0.719 0.178 Inf    -1.170   -0.2681  -4.033  0.0003
 Week 8  -0.984 0.166 Inf    -1.404   -0.5642  -5.929  <.0001
 Week 16 -1.198 0.162 Inf    -1.606   -0.7889  -7.413  <.0001
 Week 56 -1.456 0.290 Inf    -2.190   -0.7227  -5.021  <.0001

Covariance estimate used: vbeta 
Confidence level used: 0.95 
Conf-level adjustment: mvt method for 6 estimates 
P value adjustment: mvt method for 6 tests 

Now I'm trying to do the same with parameters (expecting it will pass the parameter to emmeans since it understands this class):

> parameters(emg, p_adjust = "mvt")
Week    | Marginal Means |   SE |         95% CI |     z |      p
-----------------------------------------------------------------
Week 1  |          -0.32 | 0.13 | [-0.58, -0.06] | -2.39 | 0.017 
Week 2  |          -0.63 | 0.14 | [-0.90, -0.37] | -4.64 | < .001
Week 4  |          -0.72 | 0.18 | [-1.07, -0.37] | -4.03 | < .001
Week 8  |          -0.98 | 0.17 | [-1.31, -0.66] | -5.93 | < .001
Week 16 |          -1.20 | 0.16 | [-1.51, -0.88] | -7.41 | < .001
Week 56 |          -1.46 | 0.29 | [-2.02, -0.89] | -5.02 | < .001

p-value adjustment method: mvt
Warning message:
`p_adjust` must be one of holm, hochberg, hommel, bonferroni, bh, by, fdr, none, tukey, scheffe, sidak 

OK, let's try "bh"

> parameters(emg, p_adjust = "bh")
Error in match.arg(method) : 
  'arg' should be one of “holm”, “hochberg”, “hommel”, “bonferroni”, “BH”, “BY”, “fdr”, “none”

Also doesn't work. So maybe the "BH":

> parameters(emg, p_adjust = "BH")
Week    | Marginal Means |   SE |         95% CI |     z |      p
-----------------------------------------------------------------
Week 1  |          -0.32 | 0.13 | [-0.58, -0.06] | -2.39 | 0.017 
Week 2  |          -0.63 | 0.14 | [-0.90, -0.37] | -4.64 | < .001
Week 4  |          -0.72 | 0.18 | [-1.07, -0.37] | -4.03 | < .001
Week 8  |          -0.98 | 0.17 | [-1.31, -0.66] | -5.93 | < .001
Week 16 |          -1.20 | 0.16 | [-1.51, -0.88] | -7.41 | < .001
Week 56 |          -1.46 | 0.29 | [-2.02, -0.89] | -5.02 | < .001

p-value adjustment method: Benjamini & Hochberg (1995)

This one worked.

So, to summarize:

  1. the "mvt" isn't passed to emmeans, even if parameters understand this class (but maybe it was not the goal - then I'm sorry for reporting it)
  2. but even if we pick the other options, the hint is given using all small letters

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the parameters() entry point for emmeans/emGrid objects and trace how p_adjust is validated and passed through. Compare the handling of "mvt", lowercase "bh", and uppercase "BH" with the behavior shown in the examples. Done means supported adjustment names are handled consistently and the requested emmeans adjustment behavior is preserved or clearly reported.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
analytics
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.