data.table .internal.selfref ptr is NULL case using foreach
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 30/100
Research direction
Start by running the supplied reproduction from bug.R with data.table, foreach, and doMC, then compare the %dopar% and %do% cases. Trace the .internal.selfref warning and determine whether the behavior belongs to data.table or foreach; done means the ownership and expected behavior are established with a regression test or documented resolution.
Written by the indexing model from the issue text.
Description
Hello I came across the following (bug?), I do not know if it comes from foreach or data.table or if I do something stupid.
library(data.table)
library(foreach)
library(doMC)
registerDoMC(cores = 2L)
options(datatable.verbose = TRUE)
DT <- data.table(x=1:2, y = 2:3, z = c("a", "b"))
DT_list <- split(DT, by = "z")
foreach(DT_i = DT_list) %dopar%
{
fun <- function(DT) return(DT[, c("x", "y"), with = FALSE])
res_i <- fun(DT_i)
} -> res_list
res_list[[1]][,test := 1]
$ R --vanilla -e 'source("bug.R")'
> source("bug.R")
Loading required package: iterators
Loading required package: parallel
Processing split.data.table with: x[j = list(.ll.tech.split = list(.SD)), by = "z", .SDcols = c("x", "y", "z")]
Finding groups using forderv ... 0.046s elapsed (0.573s cpu)
Finding group sizes from the positions (can be avoided to save RAM) ... 0.000s elapsed (0.000s cpu)
lapply optimization is on, j unchanged as 'list(list(.SD))'
GForce is on, left j unchanged
Old mean optimization is on, left j unchanged.
Making each group and running j (GForce FALSE) ...
memcpy contiguous groups took 0.000s for 2 groups
eval(j) took 0.000s for 2 calls
0.000s elapsed (0.002s cpu)
Finding groups using forderv ... 0.000s elapsed (0.001s cpu)
Finding group sizes from the positions (can be avoided to save RAM) ... 0.000s elapsed (0.001s cpu)
lapply optimization is on, j unchanged as 'list(paste(unlist(lapply(.SD, as.character)), collapse = "."))'
GForce is on, left j unchanged
Old mean optimization is on, left j unchanged.
Making each group and running j (GForce FALSE) ...
memcpy contiguous groups took 0.000s for 2 groups
eval(j) took 0.000s for 2 calls
0.001s elapsed (0.006s cpu)
Detected that j uses these columns: <none>
.internal.selfref ptr is NULL. This is expected and normal for a data.table loaded from disk. Please remember to always setDT() immediately after loading to prevent unexpected behavior. If this table was not loaded from disk or you've already run setDT(), please report to data.table issue tracker.
Assigning to all 1 rows
RHS_list_of_columns == false
RHS for item 1 has been duplicated because NAMED is 2, but then is being plonked. length(values)==1; length(cols)==1)
I am using dev : data.table 1.12.3 IN DEVELOPMENT built 2019-07-16 10:38:13 UTC
Note that replacing %dopar% by %do% solves the issue so it may be a foreach bug ?
Only thing I am surprised about is that it is the first time I notice this and I usee foreach and data.table all the time
- Dominant language
- R
- Stars
- 3.9k
- Forks
- 1.1k
- Avg merge
- 14h 4m
- Merged PRs (30d)
- 4
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Rdatatable/data.table
-
as.data.table() recurses without end on a survival::Surv object (or any data.frame carrying one) Open
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Rdatatable/data.table#7887 ·
-
consistency tests
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Rdatatable/data.table#7853 · 3 comments ·
-
internals
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Rdatatable/data.table#6938 · 1 comment ·
-
encoding fread
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Rdatatable/data.table#5179 · 8 comments ·
-
documentation programming
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Rdatatable/data.table#3199 · 3 comments ·
All issues in Rdatatable/data.table
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
r-lib/pkgdepends#485 · 3 comments ·
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
-
beginners blocker
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enviPathR OpenBuild Error Build OK Build Warning policies-accepted pre-review precheck-passed
Difficulty 1/5 Under an hour Newbie friendliness 84/100
Bioconductor/BiocContributions#207 · 6 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
datacarpentry/semester-biology#1255 ·