tidymodels / tidymodels/multilevelmod

`predict()` ignores random effects by default in `glmer` engine

Open
#83 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug feature
Dominant language
R
Stars
74
Forks
6
Avg merge
1h 53m
Merged PRs (30d)
1

Description

The problem

The predict() method for multilevel models fitted with lme4 engines (i.e. lmer and glmer) in multilevelmod produces unexpected predictions that differ significantly from calling predict() directly on the underlying lme4 model object.
After investigation, I found this is because the prediction configuration hardcodes re.form = NA in make_lme4_logistic_reg(), which excludes random effects from predictions. With re.form = NA, predictions only use fixed effects, resulting in predictions that differ from the underlying model's default behavior.

Expected behaviour

Users should have control over whether random effects are included in predictions. There are valid use cases for both approaches:

  1. Population-level predictions (current behavior): For generalising to new groups not in the training data
  2. Conditional predictions (with random effects): For within-person or within-group predictions, accounting for individual/group-specific deviations

When analyzing longitudinal data or clustered observations, researchers often want predictions that account for individual-specific effects (e.g., predicting a specific person's outcome at a new timepoint). The current default forces population-level predictions, which may not be appropriate for all analyses.

Proposed Solution

I suggest one of the following approaches:

  1. Change the default to re.form = NULL (which includes random effects, matching lme4's default behavior)
  2. Add a parameter to predict() that allows users to specify re.form, similar to how it works in lme4
  3. Make it configurable through the parsnip workflow, allowing users to set this preference when creating the model specification

At the very least, users should be informed that this is the default behaviour. Thank you for considering this issue. I'm happy to provide additional details or testing if needed.

Reproducible example

library(multilevelmod)
#> Loading required package: parsnip
set.seed(1234)
data(sleepstudy, package = "lme4")

mixed_model_spec <- linear_reg() %>% set_engine("lmer")

mixed_model_fit <- 
  mixed_model_spec %>% 
  fit(Reaction ~ Days + (Days | Subject), data = sleepstudy)

# Without random effects by default
no_re <- predict(mixed_model_fit, sleepstudy)

# With random effects by calling lme4::predict() directly
with_re <- predict(extract_fit_engine(mixed_model_fit))

library(ggplot2)
data.frame(
  no_re = no_re$.pred,
  with_re = with_re
) |> 
  cbind(sleepstudy) |>
  ggplot(aes(x = no_re, y = with_re)) +
  geom_point() +
  facet_wrap(vars(Subject))

Created on 2025-10-15 with reprex v2.1.1

Session info

 
sessioninfo::session_info()
#> Warning in system2("quarto", "-V", stdout = TRUE, env = paste0("TMPDIR=", :
#> running command '"quarto"
#> TMPDIR=C:/Users/u0134047/AppData/Local/Temp/RtmpK2Dzxu/fileb35456e02016 -V' had
#> status 1
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.5.1 (2025-06-13 ucrt)
#>  os       Windows 11 x64 (build 26100)
#>  system   x86_64, mingw32
#>  ui       RTerm
#>  language (EN)
#>  collate  Dutch_Netherlands.utf8
#>  ctype    Dutch_Netherlands.utf8
#>  tz       Europe/Brussels
#>  date     2025-10-15
#>  pandoc   3.6.3 @ c:\\Workdir\\MyApps\\Positron\\resources\\app\\quarto\\bin\\tools/ (via rmarkdown)
#>  quarto   NA @ C:\\Workdir\\MyApps\\RStudio\\RESOUR~1\\app\\bin\\quarto\\bin\\quarto.exe
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package       * version date (UTC) lib source
#>  boot            1.3-32  2025-08-29 [1] CRAN (R 4.5.1)
#>  cli             3.6.5   2025-04-23 [1] CRAN (R 4.5.0)
#>  curl            7.0.0   2025-08-19 [1] CRAN (R 4.5.1)
#>  dichromat       2.0-0.1 2022-05-02 [1] CRAN (R 4.4.0)
#>  digest          0.6.37  2024-08-19 [1] CRAN (R 4.4.1)
#>  dplyr           1.1.4   2023-11-17 [1] CRAN (R 4.4.1)
#>  evaluate        1.0.5   2025-08-27 [1] CRAN (R 4.5.1)
#>  farver          2.1.2   2024-05-13 [1] CRAN (R 4.4.1)
#>  fastmap         1.2.0   2024-05-15 [1] CRAN (R 4.4.1)
#>  fs              1.6.6   2025-04-12 [1] CRAN (R 4.5.0)
#>  generics        0.1.4   2025-05-09 [1] CRAN (R 4.5.0)
#>  ggplot2       * 4.0.0   2025-09-11 [1] CRAN (R 4.5.1)
#>  glue            1.8.0   2024-09-30 [1] CRAN (R 4.4.1)
#>  gtable          0.3.6   2024-10-25 [1] CRAN (R 4.4.2)
#>  hardhat         1.4.2   2025-08-20 [1] CRAN (R 4.5.1)
#>  htmltools       0.5.8.1 2024-04-04 [1] CRAN (R 4.4.1)
#>  knitr           1.50    2025-03-16 [1] CRAN (R 4.5.0)
#>  labeling        0.4.3   2023-08-29 [1] CRAN (R 4.4.0)
#>  lattice         0.22-7  2025-04-02 [1] CRAN (R 4.5.0)
#>  lifecycle       1.0.4   2023-11-07 [1] CRAN (R 4.4.1)
#>  lme4            1.1-37  2025-03-26 [1] CRAN (R 4.5.0)
#>  magrittr        2.0.4   2025-09-12 [1] CRAN (R 4.5.1)
#>  MASS            7.3-65  2025-02-28 [1] CRAN (R 4.5.0)
#>  Matrix          1.7-4   2025-08-28 [1] CRAN (R 4.5.1)
#>  minqa           1.2.8   2024-08-17 [1] CRAN (R 4.4.1)
#>  multilevelmod * 1.0.0   2022-06-17 [1] CRAN (R 4.4.1)
#>  nlme            3.1-168 2025-03-31 [1] CRAN (R 4.5.0)
#>  nloptr          2.2.1   2025-03-17 [1] CRAN (R 4.5.0)
#>  parsnip       * 1.3.3   2025-08-31 [1] CRAN (R 4.5.1)
#>  pillar          1.11.1  2025-09-17 [1] CRAN (R 4.5.1)
#>  pkgconfig       2.0.3   2019-09-22 [1] CRAN (R 4.4.1)
#>  purrr           1.1.0   2025-07-10 [1] CRAN (R 4.5.1)
#>  R6              2.6.1   2025-02-15 [1] CRAN (R 4.4.2)
#>  rbibutils       2.3     2024-10-04 [1] CRAN (R 4.4.1)
#>  RColorBrewer    1.1-3   2022-04-03 [1] CRAN (R 4.4.0)
#>  Rcpp            1.1.0   2025-07-02 [1] CRAN (R 4.5.1)
#>  Rdpack          2.6.4   2025-04-09 [1] CRAN (R 4.5.0)
#>  reformulas      0.4.1   2025-04-30 [1] CRAN (R 4.5.0)
#>  reprex          2.1.1   2024-07-06 [1] CRAN (R 4.4.1)
#>  rlang           1.1.6   2025-04-11 [1] CRAN (R 4.5.0)
#>  rmarkdown       2.30    2025-09-28 [1] CRAN (R 4.5.1)
#>  S7              0.2.0   2024-11-07 [1] CRAN (R 4.4.2)
#>  scales          1.4.0   2025-04-24 [1] CRAN (R 4.5.0)
#>  sessioninfo     1.2.3   2025-02-05 [1] CRAN (R 4.4.2)
#>  sparsevctrs     0.3.4   2025-05-25 [1] CRAN (R 4.5.0)
#>  tibble          3.3.0   2025-06-08 [1] CRAN (R 4.5.0)
#>  tidyr           1.3.1   2024-01-24 [1] CRAN (R 4.4.1)
#>  tidyselect      1.2.1   2024-03-11 [1] CRAN (R 4.4.1)
#>  vctrs           0.6.5   2023-12-01 [1] CRAN (R 4.4.1)
#>  withr           3.0.2   2024-10-28 [1] CRAN (R 4.4.2)
#>  xfun            0.53    2025-08-19 [1] CRAN (R 4.5.1)
#>  xml2            1.4.0   2025-08-20 [1] CRAN (R 4.5.1)
#>  yaml            2.3.10  2024-07-26 [1] CRAN (R 4.4.1)
#> 
#>  [1] C:/Workdir/MyApps/R-Library/4.0
#>  [2] C:/Workdir/MyApps/R/R-4.5.1/library
#>  * ── Packages attached to the search path.
#> 
#> ──────────────────────────────────────────────────────────────────────────────

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the discrepancy with the lmer and glmer examples, then inspect make_lme4_logistic_reg() where re.form = NA is hardcoded and compare it with lme4::predict() defaults. Done means the chosen behavior for random effects is implemented consistently, users can control it or are clearly informed, and predictions are covered for the affected engines.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.