bug with get() when grouping keyed DT

Start by running the minimal reproducible example involving setkey(), get(), grouped max(), and split.data.table(). Trace the grouping and GForce paths used when the result is keyed by group; done means grouped summaries and split() produce one result per actual group without duplicated levels.

The original data.table that had the issue was obtained via a long series of data manipulation in a real-word data analysis task starting from proprietary data, and the entire process is somewhat complex. Previously I could not reproduce the issue on toy examples so I uploaded the particular problematic data.table to a github repo, but I have since found an artificial minimal reproducible example, so the content below has been updated.

Minimal reproducible example

library(data.table)

tmp <- data.table(id=rep(letters[1:4], each=10), group=rep(letters[1:2], each=10), x1=runif(40))
tmp1 <- data.table(id=letters[1:4], y1=runif(4))
dat <- merge.data.table(tmp, tmp1, by="id") # `dat` has key <id>
dat1 <- dat[, .(x=get("x1"), group, id)] # have to involve `get`, and have to include id here, otherwise no bug; on print `dat1` is displayed to have key <group>

str(dat1)
# Classes ‘data.table’ and 'data.frame':        40 obs. of  3 variables:
#  $ x    : num  0.791 0.486 0.386 0.168 0.689 ...
#  $ group: chr  "a" "a" "a" "a" ...
#  $ id   : chr  "a" "a" "a" "a" ...
#  - attr(*, ".internal.selfref")=<externalptr>
#  - attr(*, "sorted")= chr "group"

dat1[, max(x), by=group, verbose=TRUE] # bug: duplicated group levels and incorrect summaries
# Detected that j uses these columns: [x]
# Finding groups using uniqlist on key ... 0.000s elapsed (0.000s cpu)
# Finding group sizes from the positions (can be avoided to save RAM) ... 0.001s elapsed (0.000s cpu)
# lapply optimization is on, j unchanged as 'max(x)'
# GForce optimized j to 'gmax(x)' (see ?GForce)
# Making each group and running j (GForce TRUE) ... gforce initial population of grp took 0.000
# gforce assign high and low took 0.000
# gforce eval took 0.000
# 0.000s elapsed (0.001s cpu)
# Key: <group>
#     group        V1
#    <char>     <num>
# 1:      a 0.8584829
# 2:      b 0.9814620
# 3:      a 0.8684479
# 4:      b 0.9429159

# also, e.g. split
str(split(dat1, by="group", verbose=TRUE))
# Processing split.data.table with: x[j = list(.ll.tech.split = list(.SD), .ll.tech.split.names = paste(lapply(.BY, as.character), collapse = ".")), by = "group", keyby = FALSE, .SDcols = c("x", "group", "id")]
# List of 4
#  $ a:Classes ‘data.table’ and 'data.frame':   10 obs. of  3 variables:
#   ..$ x    : num [1:10] 0.791 0.486 0.386 0.168 0.689 ...
#   ..$ group: chr [1:10] "a" "a" "a" "a" ...
#   ..$ id   : chr [1:10] "a" "a" "a" "a" ...
#   ..- attr(*, ".internal.selfref")=<externalptr>
#   ..- attr(*, "sorted")= chr "group"
#  $ b:Classes ‘data.table’ and 'data.frame':   10 obs. of  3 variables:
#   ..$ x    : num [1:10] 0.479 0.751 0.832 0.14 0.981 ...
#   ..$ group: chr [1:10] "b" "b" "b" "b" ...
#   ..$ id   : chr [1:10] "b" "b" "b" "b" ...
#   ..- attr(*, ".internal.selfref")=<externalptr>
#   ..- attr(*, "sorted")= chr "group"
#  $ a:Classes ‘data.table’ and 'data.frame':   10 obs. of  3 variables:
#   ..$ x    : num [1:10] 0.7827 0.0303 0.0451 0.6802 0.2384 ...
#   ..$ group: chr [1:10] "a" "a" "a" "a" ...
#   ..$ id   : chr [1:10] "c" "c" "c" "c" ...
#   ..- attr(*, ".internal.selfref")=<externalptr>
#   ..- attr(*, "sorted")= chr "group"
#  $ b:Classes ‘data.table’ and 'data.frame':   10 obs. of  3 variables:
#   ..$ x    : num [1:10] 0.457 0.813 0.49 0.099 0.853 ...
#   ..$ group: chr [1:10] "b" "b" "b" "b" ...
#   ..$ id   : chr [1:10] "b" "b" "b" "b" ...
#   ..- attr(*, ".internal.selfref")=<externalptr>
#   ..- attr(*, "sorted")= chr "group"
#  $ a:Classes ‘data.table’ and 'data.frame':   10 obs. of  3 variables:
#   ..$ x    : num [1:10] 0.7827 0.0303 0.0451 0.6802 0.2384 ...
#   ..$ group: chr [1:10] "a" "a" "a" "a" ...
#   ..$ id   : chr [1:10] "c" "c" "c" "c" ...
#   ..- attr(*, ".internal.selfref")=<externalptr>
#   ..- attr(*, "sorted")= chr "group"
#  $ b:Classes ‘data.table’ and 'data.frame':   10 obs. of  3 variables:
#   ..$ x    : num [1:10] 0.457 0.813 0.49 0.099 0.853 ...
#   ..$ group: chr [1:10] "b" "b" "b" "b" ...
#   ..$ id   : chr [1:10] "d" "d" "d" "d" ...
#   ..- attr(*, ".internal.selfref")=<externalptr>
#   ..- attr(*, "sorted")= chr "group"

# this has no bug, since `dat` has no key; if I do `setkey(dat, id)` after the join, the bug will appear
dat <- tmp[tmp1, on="id"] # no key
dat1 <- dat[, .(x=get("x1"), group, id)] # no key
dat1[, max(x), by=group, verbose=TRUE] # no bug
# Detected that j uses these columns: [x]
# Finding groups using forderv ... forderReuseSorting: opt not possible: is.data.table(DT)=0, sortGroups=0, all1(ascArg)=1
# forder.c received 40 rows and 1 columns
# forderReuseSorting: opt=0, took 0.000s
# 0.000s elapsed (0.001s cpu)
# Finding group sizes from the positions (can be avoided to save RAM) ... 0.000s elapsed (0.000s cpu)
# Getting back original order ... forderReuseSorting: opt not possible: is.data.table(DT)=0, sortGroups=1, all1(ascArg)=1
# forder.c received a vector type 'integer' length 2
# forderReuseSorting: opt=0, took 0.000s
# 0.000s elapsed (0.000s cpu)
# lapply optimization is on, j unchanged as 'max(x)'
# GForce optimized j to 'gmax(x)' (see ?GForce)
# Making each group and running j (GForce TRUE) ... gforce initial population of grp took 0.000
# gforce assign high and low took 0.000
# gforce eval took 0.000
# 0.000s elapsed (0.001s cpu)
#     group        V1
#    <char>     <num>
# 1:      a 0.8684479
# 2:      b 0.9814620

# even more directly
dat <- data.table(id=rep(letters[1:4], each=10), group=rep(letters[1:2], each=10), x1=runif(40))
setkey(dat, id)
dat1 <- dat[, .(x=get("x1"), group, id)] # have to involve `get`, and have to include id here, otherwise no bug; on print `dat1` is displayed to have key <group>
dat1[, max(x), by=group] # bug
# Key: <group>
#     group        V1
#    <char>     <num>
# 1:      a 0.9236200
# 2:      b 0.7971480
# 3:      a 0.9688696
# 4:      b 0.9933065

Output of sessionInfo()

R version 4.4.1 (2024-06-14)
Platform: x86_64-pc-linux-gnu
Running under: CentOS Linux 7 (Core)

Matrix products: default
BLAS/LAPACK: /SFS/product/R/4.4.1/linux-centos7-x86_64/lib64/R/lib/libmkl_rt.so.2;  LAPACK version 3.10.1

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8
 [8] LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

time zone: America/New_York
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] nvimcom_0.9-167   magrittr_2.0.3    stringr_1.5.1     data.table_1.16.4

loaded via a namespace (and not attached):
[1] compiler_4.4.1  cli_3.6.3       tools_4.4.1     glue_1.8.0      stringi_1.8.4   lifecycle_1.0.4 rlang_1.1.5