Error on modifying by reference with `data.table::set()` in the context of `future.apply::future_apply()` or `furrr::future_map()`

Reproduce the supplied R example with lapply(), future_lapply(), and future_walk(), using the daeqtlr installation and data files described in the issue. Start at data.table::set(), comparing its behavior after serialization in the parallel workers and the suggested setDT() or setalloccol() calls; done means the parallel examples no longer fail or the issue is shown to belong to future.apply or furrr.

Hi,

First of all, let me thank you for the development amazing {data.table} package.

My case is that I have a list of data tables that I am trying to modify by reference with data.table::set() inside a loop using future.apply::future_apply() and furrr::future_walk()/furrr::future_map().

However I am getting an error when using future.apply::future_apply() or furrr::future_walk()/furrr::future_map(). It works fine with lapply() although.

I am not sure the problem is with the {data.table} package itself... I will post this same issue in {furrr} and {future.apply} Issues, and link it here.

The error is:

Error in data.table::set(snp_pairs, i = i, j = col, value = df[[col]]) : 
  This data.table has either been loaded from disk (e.g. using readRDS()/load()) or constructed manually (e.g. using structure()). Please run setDT() or setalloccol() on it first (to pre-allocate space for new columns) before assigning by reference to it.

You will need to install {daeqtlr} first:

remotes::install_github("maialab/daeqtlr")

library(future.apply)
library(furrr)
# For now install from https://github.com/maialab/daeqtlr
library(daeqtlr)

plan(multisession)

snp_pairs <- read_snp_pairs(file = daeqtlr_example("snp_pairs.csv"))
zygosity <- read_snp_zygosity(file = daeqtlr_example("zygosity.csv"))
ae <- read_ae_ratios(file = daeqtlr_example("ae.csv"))

no_cores <- 6L
indices <- seq_len(nrow(snp_pairs))
partitioning_factor <- sort((indices)%%no_cores) + 1
snp_pairs_lst1 <- split(snp_pairs, partitioning_factor)
snp_pairs_lst2 <- split(snp_pairs, partitioning_factor)
snp_pairs_lst3 <- split(snp_pairs, partitioning_factor)

for( i in seq_along(snp_pairs_lst1)) {
  data.table::setkeyv(snp_pairs_lst1[[i]], 'dae_snp')
  data.table::setkeyv(snp_pairs_lst2[[i]], 'dae_snp')
  data.table::setkeyv(snp_pairs_lst3[[i]], 'dae_snp')
}

# Runs fine without errors.
lapply(snp_pairs_lst1,
              FUN = daeqtl_mapping,
              zygosity = zygosity,
              ae = ae)

# Fails with error:
# 
# Error in data.table::set(snp_pairs, i = i, j = col, value =
# df[[col]]) : This data.table has either been loaded from disk (e.g. using
# readRDS()/load()) or constructed manually (e.g. using structure()). Please run
# setDT() or setalloccol() on it first (to pre-allocate space for new columns)
# before assigning by reference to it.
future_lapply(snp_pairs_lst2,
              FUN = daeqtl_mapping,
              zygosity = zygosity,
              ae = ae)

# Fails with the same error as `future_lapply`
# It won't work with `future_map` either.
future_walk(snp_pairs_lst3,
              .f = daeqtl_mapping,
              zygosity = zygosity,
              ae = ae)