Optimise dcast.data.table to reduce peak memory usage (currently: reaches 4x table size)

I am using data.table's melt and dcast function to melt a wide table, do some transformation and cast it back to a tidy format.

The Issue

I tested with a small file of ~1GB and was very surprised when R ran out of memory on my i7, 16GB machine. I investigated further and realise that dcast.data.table hit a peak memory of ~4x the long-format file's size.

Running my sample code below, the memory requirements for the process are as follow:

• Original table = 67mb (screenshot 1)
• Melted long table = 184mb (incremental peak memory: 269-67 = 202mb)
• After transformation = 214mb (screenshot 2)
• Final table after dcast = 123mb (incremental peak memory: 935-214 = 721mb) (screenshot 2)

I added reshape2's dcast for comparison:

• Incremental peak memory: 834-214 = 620mb

My Request

Can memory usage for dcast.data.table be optimised further?

dcast.data.table is certainly faster than its counterpart but it appears there is some memory overhead. The 1GB file I tested originally is only ~30% of the full dataset, so a dcast.data.table that is more frugal is very much appreciated =)

Sample Data and Code

library( data.table )
library( reshape2 )
library( stringr )

gc( reset = T )
set.seed(1)
nrow <- 5e5
ColNames <- c( "IDCol1", "IDCol2", "IDCol3", 
               paste0( "A_dd", 1:8 ), paste0( "B_dd", 1:8 ) )

# Create DT
dt <- as.data.table( setNames( c(
  replicate( 2, sample( state.name, nrow, replace = T ), simplify = F ), 
  replicate( 1, 1:nrow, simplify = F ), 
  replicate( 16, round( runif( nrow, 1, 30 ), 2), simplify = F ) ), 
    ColNames ) )
tables()
gc( reset = T )

# Melt dt
L_dt <- melt( dt, id = c( "IDCol1", "IDCol2", "IDCol3" ) )
rm( dt )
gc()

tables()
gc( reset = T )

# Transform dt
L_dt[ , NewCol1 := as.integer( str_extract( L_dt$variable, "[0-9]$" ) ) ]
L_dt[ , variable := str_replace( L_dt$variable, "[0-9]$", "" ) ]
tables()
gc( reset = T )

# Using data.table to re-cast L_dt to W_dt
W_dt1 <- dcast.data.table( L_dt, IDCol1+IDCol2+IDCol3+NewCol1 ~ variable, 
                          value.var = "value" )
gc()
tables()
rm( W_dt1 )
gc( reset = T )


# Using Reshape2 to re-cast L_dt to W_dt
W_dt2 <- dcast( L_dt, IDCol1+IDCol2+IDCol3+NewCol1 ~ variable, 
                          value.var = "value" )
gc()