futureverse / futureverse/future.apply
Non-blocking, local evaluation of future_lapply()?
- Dominant language
- R
- Stars
- 218
- Forks
- 20
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 1
Description
Is future_lapply() intended to be non-blocking? I thought it was, but it's not working for me.
Here is a minimal reproducible example (with a simple function that serves no purpose except to take ~10 seconds to evaluate on my machine so I could test blocking and multicore behavior). I'm running Linux, so my understanding is that multiprocess here implies multicore. (Output shown as comments.)
```r
# future_lapply() blocks, even in multiprocess. You can see that resolved()
# does not get evaluated until future_lapply() has finished.
# But it successfully distributes this across 2 cores.
plan(multiprocess)
date()
### [1] "Fri May 24 14:29:51 2019"
a <- future_lapply(rep(50000000, 2), function(i) rnorm(i)*rnorm(i))
resolved(futureOf(a))
### Error: Future (‘a’) not found in environment ‘R_GlobalEnv’: ‘a’
date()
### [1] "Fri May 24 14:30:03 2019"
head(a[[1]])
### [1] -1.2233054 0.1918043 -0.4650852 0.5335259 -0.2493615 -0.8267408
date()
### [1] "Fri May 24 14:30:03 2019"
```
Note that I get an error when trying to call `resolved(futureOf(a))`, because `a` has already been resolved before it gets called, and no future exists because it was implicit. The calls to `date()` are in there to show that it blocked for 12 seconds while it was evaluating `future_lapply()`.
Based on your response in #1, I tried assigning `future_lapply` as an implicit future and using nested multiprocess evaluation (though that was intended for someone running SGE). This also blocks, and now `future_lapply()` is evaluated sequentially, not on multiple cores. I watched process allocation happening, but you can see that it now blocks for twice as long: 24 seconds.
```r
# Try nesting it in an implicit future call. Still blocks. But now this gets
# evaluated sequentially rather than distributed across 2 cores.
plan(list(multiprocess, multiprocess))
date()
### "Fri May 24 14:37:15 2019"
a %<-% future_lapply(rep(50000000, 2), function(i) rnorm(i)*rnorm(i))
resolved(futureOf(a))
### [1] FALSE
date()
### [1] "Fri May 24 14:37:15 2019"
head(a[[1]])
### [1] -0.9747142 -0.1586670 -0.1039924 4.5885303 -0.4779900 0.3339059
date()
### [1] "Fri May 24 14:37:39 2019"
```
I ran into this issue because I'm trying to switch from `mclapply` to `future_lapply` (for the great parallel RNG!), and I do get non-blocking behavior using an implicit future with `mclapply` (`resolved()` and `date()` are both executed immediately after the `mclapply` call without blocking):
```r
# This works as expected: Setting mc.cores explicitly does distribute across
# multiple cores, and it's non-blocking.
library(parallel)
plan(multiprocess)
date()
### [1] "Fri May 24 14:51:31 2019"
a %<-% mclapply(rep(50000000, 2), function(i) rnorm(i)*rnorm(i), mc.cores=2)
resolved(futureOf(a))
### [1] FALSE
date()
### [1] "Fri May 24 14:51:31 2019"
head(a[[1]])
### [1] 0.968440961 -0.015869658 0.321415096 -0.609809739 0.005155251
date()
### [1] "Fri May 24 14:51:44 2019"
```
Incidentally, if I replace the call to explicitly set `mc.cores=2` with `mc.cores=future::availableCores()`, I still get non-blocking behavior, but now `mclapply` gets executed sequentially instead of being distributed across cores. (If I run `mc.cores=future::availableCores()` I get `16`.) I'm not sure if this is a bug, and I didn't explore it thoroughly, but it's not what I expected.
Thanks so much for your help and for all your work to bring R into the future!
Contributor guide
Assessment
This issue has not been assessed yet.