Invalid .internal.selfref detected when a data.table is set as an attribute with setattr

Start by reproducing the two examples with R and data.table, comparing attr<-, setattr, attr(), and := behavior. Trace the setattr and attribute-handling paths to identify why the data.table self-reference changes. Done means the reported attribute mutation behavior is corrected without producing the invalid .internal.selfref warning.

If I have an object (data.table or not) and I add a data.table attribute to it via attr<-, the address of the OBJECT changes, but the address of the ATTRIBUTE does not. I can later extract the data.table attribute, and modifications to it will also modify the attribute.

x <- 'x'
dt <- data.table(a = 1)
tracemem(dt)
#> [1] "<0x39b9bd0>"
tracemem(x)
#> [1] "<0x3203148>"
attr(x, 'dt') <- dt
#> tracemem[0x3203148 -> 0x224b840]: eval
dt2 <- attr(x, 'dt')
dt2[, b := 1]
dt2[]
#>    a b
#> 1: 1 1
attr(x, 'dt')
#>    a b
#> 1: 1 1

But if I add a data.table attribute to it via setattr, the address of the OBJECT does not change, whereas the address of the ATTRIBUTE does. If I later extract the attribute, it will be de-referenced:

x <- 'x'
dt <- data.table(a = 1)
tracemem(dt)
#> [1] "<0x46d3dc0>"
tracemem(x)
#> [1] "<0x3207768>"
setattr(x, 'dt', dt)
#> tracemem[0x46d3dc0 -> 0x432dfc0]: setattr
dt2 <- attr(x, 'dt')
dt2[, b := 1]
#> Warning in `[.data.table`(dt2, , `:=`(b, 1)): Invalid .internal.selfref
#> detected and fixed by taking a (shallow) copy of the data.table so
#> that := can add this new column by reference. At an earlier point, this
#> data.table has been copied by R (or was created manually using structure()
#> or similar). Avoid names<- and attr<- which in R currently (and oddly)
#> may copy the whole data.table. Use set* syntax instead to avoid copying: ?
#> set, ?setnames and ?setattr. If this message doesn't help, please report
#> your use case to the data.table issue tracker so the root cause can be
#> fixed or this message improved.
dt2[]
#>    a b
#> 1: 1 1
attr(x, 'dt')
#>    a
#> 1: 1

Note that in this second example I used setattr, not attr<-, and yet seemingly R copied the data.table somewhere. The message didn't help, so here I am.

sessionInfo:

R version 3.5.2 (2018-12-20)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.5 LTS

Matrix products: default
BLAS: /usr/lib/libblas/libblas.so.3.6.0
LAPACK: /usr/lib/lapack/liblapack.so.3.6.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] data.table_1.12.2

loaded via a namespace (and not attached):
[1] compiler_3.5.2 tools_3.5.2    yaml_2.2.0