Column selection (incl. select-on-join) inefficient when j is list()

Reproduce the column-selection and select-on-join benchmarks in the issue, then start at [.data.table and the handling of list()/.() in j. Compare the allocation and timing results with standard-evaluation and data.frame() alternatives; done means the reported extra copy and performance hit are addressed or clearly characterized by regression benchmarks.

I understand that column subsetting with [.data.table makes a deep copy...

library(data.table)
dt <- data.table(rep(1L,1e8L)) # 380MB
addr <- address(dt$V1)
dt[,address(V1)]      == addr # TRUE
address(dt[,"V1"]$V1) == addr # FALSE
address(dt[,V1])      == addr # FALSE

...and that this behaviour differs from base R [.data.frame, which returns a shallow copy:

setDF(dt)
address(dt[,"V1"]) == addr # TRUE 
setDT(dt)

But what is happening here? Selection and conversion to a data.table by means of list() (case 4) takes double the time and memory of the standard evaluation equivalent (case 3), the same as selecting the column twice (case 5).

  expression                  median mem_alloc n_itr
  <bch:expr>                <bch:tm> <bch:byt> <int>
1 "dt"                        50.5ns        0B    10
2 "dt[, V1]"                  86.3ms     382MB     8
3 "dt[, \"V1\"]"              92.6ms     381MB     9
4 "dt[, list(V1)]"           180.2ms     763MB     6
5 "dt[, c(\"V1\", \"V1\")]"  183.8ms     763MB     8

It seems that using list() and letting data.table handle the conversion entails an intermediate copy, whereas there is no extra copy if we call data.table() (or as.data.table(list())) explicitly:

 expression                      median mem_alloc n_itr
 <bch:expr>                    <bch:tm> <bch:byt> <int>
1 dt[, data.table(V1)]            85.6ms     382MB     8
2 dt[, as.data.table(list(V1))]   85.6ms     382MB     9

Extracting columns from a data.table as a new object is a bit pointless, but a similar issue seems to affect select-on-join, which is a very common operation, e.g. when creating a new data.table with a subset of the columns of two joined tables.

n <- 1e6L
dt1 <- data.table(id=1:n,v1=1L)
dt2 <- data.table(id=1:n,v2=2L)

  expression                                           median mem_alloc n_itr
  <bch:expr>                                         <bch:tm> <bch:byt> <int>
1 "dt1[dt2, on = \"id\"]"                              65.4ms    34.4MB    10
2 "dt1[dt2, on = \"id\", c(\"id\", \"v1\", \"v2\")]"   64.3ms    34.4MB    10
3 "dt1[dt2, on = \"id\", list(id, v1, v2)]"              72ms    49.7MB     9

Here, the performance of list() is actually worse than chaining a redundant copy of the joined columns (which have total size 12MB):

  expression                                             median mem_alloc n_itr
  <bch:expr>                                           <bch:tm> <bch:byt> <int>
1 "dt1[dt2, on = \"id\"][, c(\"id\", \"v1\", \"v2\")]"   67.4ms    45.9MB    10

Using data.table() instead of list() doesn't help much in this case (it mirrors the redundant copy case), but data.frame() provides a way round:

  expression                                               median mem_alloc n_itr
  <bch:expr>                                             <bch:tm> <bch:byt> <int>
1 "dt1[dt2, on = \"id\", data.table(id, v1, v2)]"          68.5ms    45.9MB    10
2 "dt1[dt2, on = \"id\", setDT(data.frame(id, v1, v2))]"   66.5ms    34.5MB    10

Is this performance hit from using list()/.() with columns-as-variables recognised/accepted? It has come as a bit of shock to me as a mainly interactive user who has always assumed it was standard and efficient. NSE is very convenient generally, and in particular it is much more concise and readable than using standard evaluation plus setnames() when renaming columns in select-on-join (.(id, foo = x.bar)). But it's not hard to cook up examples where it almost doubles the execution time and memory footprint of a join, e.g.:

n1 <- 1e8L; n2 <- 1e6L
dt1 <- data.table(id=rep(1:n2,each=n1/n2),v1=1L,w1=1L,x1=1L)
dt2 <- data.table(id=1:n2,v2=1L,w2=1L,x2=1L)
setkey(dt1,id)
setkey(dt2,id)

expression                                               median mem_alloc n_itr
 <bch:expr>                                               <bch:> <bch:byt> <int>
1 "dt1[dt2, on = .(id), c(\"id\", \"v1\", \"w1\", \"x1\",…  1.73s    3.37GB     3
2 "dt1[dt2, on = .(id), list(id, v1, w1, x1, v2, w2, x2)]"  3.01s    6.35GB     3
3 "dt1[dt2, on = .(id), setDT(data.frame(id, v1, w1, x1, …  1.82s    3.37GB     3
4 "setDT(dt1[dt2, on = .(id), data.frame(id, v1, w1, x1, …  1.81s    3.37GB     3

Assuming this is not news, the next question is what to do instead if one still wants to use columns-as-variables. Is this idea of data.frame() plus setDT() a safe and sensible approach? I've noticed that using setDT(data.frame()) inside j (case 3), although neater than calling setDT() on the joined result (case 4), is slow in certain other cases when both tables are keyed (I can add examples if relevant).