.SD in grouped operations causes update() to refit on wrong data subset

Start by running the reproducible data.table and lme4 example in the issue, then inspect how .SD is stored in the model formula environments during grouped and ungrouped operations. Compare the environments for m1 and m2 and verify that update() refits each model on its original subset, including the expected group-specific data.

I often fit models to subsets within data.tables, similar to Section 3.3 in the .SD vignette. However, is the following behavior intended?

A model fit m1 on the subset Grp == "A" is identical to the respective grouped model fit m2, as expected. Yet, a trivial update() call yields vastly different results, which did surprise me a little.

library(data.table)
library(lme4)

set.seed(2026)

dat <- CJ(Grp = c("A", "B"),
           ID = factor(1:8),
           Time = factor(c("T1", "T2", "T3", "T4")))
dat[, y := 3 * (Time == "T2") + rnorm(20, 3)[as.integer(ID)] + rnorm(.N, sd = 3)]

# Fit models using .SD --- ungrouped subset vs. grouped
m1 <- dat[Grp == "A", .(model = .(lmer(y ~ Time + (1 | ID))))]$model[[1]]
m2 <- dat[, .(model = .(lmer(y ~ Time + (1 | ID)))), by = Grp]$model[[1]]

all.equal(m1, m2) # equal as expected
all.equal(update(m1), update(m2)) # different results

As far as I understand, this issue stems from how environments are evaluated:

sd_m1 <- get(".SD", environment(m1@call$formula))
sd_m2 <- get(".SD", environment(m2@call$formula))

all.equal(sd_m1[, y], dat[Grp == "A", y]) # TRUE
all.equal(sd_m2[, y], dat[Grp == "B", y]) # TRUE, but should be Grp == "A"!

I would greatly appreciate it if there were (or someone could point me to) an elegant way to ensure that .SD retains the correct group-specific subset when used with a grouping variable :)

> sessioninfo::session_info()
─ Session info ────────────────────────────────────────────────────────────────────────────
 setting  value
 version  R version 4.6.0 (2026-04-24 ucrt)
 os       Windows 11 x64 (build 26100)
 system   x86_64, mingw32
 ui       Positron
 language (EN)
 collate  German_Germany.utf8
 ctype    German_Germany.utf8
 tz       Europe/Berlin
 date     2026-07-23
 pandoc   NA

─ Packages ────────────────────────────────────────────────────────────────────────────────
 package     * version   date (UTC) lib source
 boot          1.3-32    2025-08-29 [2] CRAN (R 4.6.0)
 cli           3.6.6     2026-04-09 [1] CRAN (R 4.6.0)
 data.table  * 1.18.4    2026-05-06 [1] CRAN (R 4.6.1)
 lattice       0.22-9    2026-02-09 [2] CRAN (R 4.6.0)
 lme4        * 2.0-1     2026-03-05 [1] CRAN (R 4.6.1)
 MASS          7.3-65    2025-02-28 [2] CRAN (R 4.6.0)
 Matrix      * 1.7-5     2026-03-21 [2] CRAN (R 4.6.0)
 minqa         1.2.8     2024-08-17 [1] CRAN (R 4.6.1)
 nlme          3.1-169   2026-03-27 [2] CRAN (R 4.6.0)
 nloptr        2.2.1     2025-03-17 [1] CRAN (R 4.6.1)
 rbibutils     2.4.1     2026-01-21 [1] CRAN (R 4.6.1)
 Rcpp          1.1.1-1.1 2026-04-24 [1] CRAN (R 4.6.0)
 Rdpack        2.6.6     2026-02-08 [1] CRAN (R 4.6.1)
 reformulas    0.4.4     2026-02-02 [1] CRAN (R 4.6.1)
 rlang         1.2.0     2026-04-06 [1] CRAN (R 4.6.0)
 sessioninfo   1.2.4     2026-06-04 [1] CRAN (R 4.6.0)