Unexpected .GRP behaviour during update-by-reference with filter, while referring to the same data.table in j

Start by running the minimal reproducible example using the grouped update-by-reference expression with .GRP and X[!is.na(val), max(val)]. Trace the .GRP and update-by-reference behavior involved, then confirm the case produces the expected values 2, 3, and 4 and add a regression test for the demonstrated behavior.

I understand that .GRP should produce a sequence of integers that are a group counter, where group 1 is 1, group 2 is 2, etc.

When doing an update-by-reference with filter, while referring to the data.table in j, I get unexpected behaviour.

Minimal reproducible example

library(data.table)

X <- data.table(
  grp1 = c("a", "a", "a", "b")
  , grp2 = c("c", "c", "d", "d")
)
X[, .GRP, grp1] # as expected
X[, .GRP, grp2] # as expected
X[, .GRP, .(grp1, grp2)] # as expected

X[, val := .GRP, .(grp1, grp2)] 
X # as expected

# reset the scenario
X[, val := NULL]

# this is a simplified example of my actual application
# the `val` column has been partially initialised, and now I need 
# to set the rest of the val column
X[1, val := 1]
X[is.na(val)
  , val := .GRP + X[!is.na(val), max(val)]
  , .(grp1, grp2)] 
X # unexpected values of .GRP. Would expect 2,3,4, but get 2,4,7

# reset the scenario
X[, val := NULL]
X[1, val := 1] 
X[is.na(val)
  , val := .GRP + 1
  , .(grp1, grp2)]
X # expected values. 

So it is something to do with X[!is.na(val), max(val)] code in the j component?

Output of sessionInfo()

> sessionInfo()
R version 4.5.1 (2025-06-13)
Platform: x86_64-apple-darwin20
Running under: macOS Sequoia 15.6.1

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib 
LAPACK: /Library/Frameworks/R.framework/Versions/4.5-x86_64/Resources/lib/libRlapack.dylib;  LAPACK version 3.12.1

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

time zone: Australia/Melbourne
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] data.table_1.17.8

loaded via a namespace (and not attached):
 [1] compiler_4.5.1    assertthat_0.2.1  rprojroot_2.1.0   tools_4.5.1       secret_1.1.0      rstudioapi_0.17.1 curl_7.0.0       
 [8] jsonlite_2.0.0    openssl_2.3.3     askpass_1.2.1