futureverse / futureverse/future
nested futures that doesn't terminate
- Dominant language
- R
- Stars
- 1k
- Forks
- 92
- PR merge metrics
- No merged PRs in 30d
Description
I am having issues with nested `future` calls.
Here I'm using `mlr3` and `furrr`, both using `future` package. When using them both together I'm having issues to have the parallelized calls to terminate.
```r
library(magrittr)
library(mlr3)
library(mlr3learners)
dtas_lst <-
list(
dta1 = iris %>% dplyr::select(-Species) %>% utils::head(75),
dta2 = iris %>% dplyr::select(-Species) %>% utils::tail(75)
)
run_xgb <- function(dta, target_column = "Sepal.Length") {
ttsk <- mlr3::TaskRegr$new("ttsk", backend = dta, target = target_column)
learner <-
mlr3::lrn(
"regr.xgboost",
objective = "reg:linear",
eval_metric = "rmse",
nrounds = 100,
verbose = 0
)
learner$train(ttsk)
learner$predict(ttsk)$response
}
future::plan(list(future::sequential))
furrr::future_map(dtas_lst, run_xgb, .progress = TRUE) # works!
future::plan(list(future::multiprocess, future::sequential))
furrr::future_map(dtas_lst, run_xgb, .progress = TRUE) # doesn't terminate
```
I've had cases where the parallel runs do work once during a session, but then not terminating on a rerun.
Sometimes when closing down the session (after manually terminating the parallel run), I get the below message
```r
# Error while shutting down parallel: unable to terminate some child processes
```
Upon googling this, I found that I could do the following check.
```r
parallel::mcparallel(scan(n = 1, quiet = TRUE))
# $pid
# [1] 32763
# $fd
# [1] 21 24
# attr(,"class")
# [1] "parallelJob" "childProcess" "process"
```
Does anyone have any solutions to this? Is this perhaps a bug? Related to `future` or perhaps `parallel` or other package? I was having the same issue when I was running the `mlr` package.
```r
> Sys.info()
sysname
"Linux"
release
"4.15.0-66-generic"
version
"#75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019"
...
machine
"x86_64"
...
```
Contributor guide
Assessment
This issue has not been assessed yet.