futureverse / futureverse/future.apply

Possible bug in loading packages

Open
#42 2 comments 1 reaction 0 assignees View on GitHub
feature request for-future-pkg
Dominant language
R
Stars
218
Forks
20
Avg merge
2d 9h
Merged PRs (30d)
1

Description

This was a bit hard to track down... Here goes.

`data.table` and `bit` both have `setattr` functions. But the annoying thing about `bit:setattr` is that it returns NULL. `data.table::setattr` returns the input object with the attribute modified invisibly. So, if you were to write a code like:

```r
y <- setattr(18004L, "class", "Date") # today's date
# [1] "2019-04-18"
```

(I'm not saying this is how one should go about it, but there are other cases where we need to set an attribute and assign the result to an object.)

In this case, depending on what `setattr` we've, it'll return the right expected result or `NULL`.

With this, consider this code:

```r
require(parallel)
require(doSNOW)
require(foreach)
require(future)
require(future.apply)
require(data.table)
foo <- function(dt) {
cat(sprintf("[%s] %s", Sys.getpid(), capture.output(environment(setattr))), sep="\n")
setattr(dt, "key", "val")
}
nodes <- 5L
cl <- future::makeClusterPSOCK(nodes)
plan(cluster, workers=cl, persistent=TRUE)
dt <- data.table(x=1, y=2)

ans <- values(
lapply(seq_len(nodes), function(node) {
future({foo(dt)}, packages=c("bit64", "data.table"))
})
)
# [15468]
# [9808]
# [15016]
# [23496]
# [22312]
```

I've created a `data.table` in the local environment (just 1 for simplicity) and am calling a function that sets the attribute in parallel. Of course this function is terribly simplified as well.

Now, the way this works (as expected), is to *first* load `bit64` first and `data.table` next and then look for functions in `foo` and possibly load more packages and then run `foo()` (AFAICT).

And this works fine as you can see from the output. If you were to check `ans`, you'd get the data.table with their attributes set.

----

Restart session (IMPORTANT). Now, with everything else remaining intact, if instead of running `values(...)`, I use `future_lapply`:

```r
ans <- future_lapply(cl, function(node) foo(dt), future.packages=c("bit64", "data.table"))
> ans <- future_lapply(cl, function(node) foo(dt), future.packages=c("bit64", "data.table"))
# [11976]
# [23212]
# [11732]
# [22964]
# [5748]
> ans
# [[1]]
# NULL
#
# [[2]]
# NULL
#
# [[3]]
# NULL
#
# [[4]]
# NULL
#
# [[5]]
# NULL
```

Note how `setattr` refers to `bit` package.. I think this is because the packages get loaded *after* assessing `setattr` is used in `foo` and `data.table` from local env has a function called `setattr` and therefore gets loaded first followed by all packages in `future.packages`?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.