futureverse / futureverse/future

nested futures that doesn't terminate

Open
#348 9 comments 0 reactions 0 assignees View on GitHub
cannot-parallelize-bc-non-exportable-object documentation
Dominant language
R
Stars
1k
Forks
92
PR merge metrics
No merged PRs in 30d

Description

I am having issues with nested `future` calls.

Here I'm using `mlr3` and `furrr`, both using `future` package. When using them both together I'm having issues to have the parallelized calls to terminate.

```r
library(magrittr)
library(mlr3)
library(mlr3learners)
dtas_lst <-
list(
dta1 = iris %>% dplyr::select(-Species) %>% utils::head(75),
dta2 = iris %>% dplyr::select(-Species) %>% utils::tail(75)
)

run_xgb <- function(dta, target_column = "Sepal.Length") {
ttsk <- mlr3::TaskRegr$new("ttsk", backend = dta, target = target_column)
learner <-
mlr3::lrn(
"regr.xgboost",
objective = "reg:linear",
eval_metric = "rmse",
nrounds = 100,
verbose = 0
)
learner$train(ttsk)
learner$predict(ttsk)$response
}
future::plan(list(future::sequential))
furrr::future_map(dtas_lst, run_xgb, .progress = TRUE) # works!
future::plan(list(future::multiprocess, future::sequential))
furrr::future_map(dtas_lst, run_xgb, .progress = TRUE) # doesn't terminate
```

I've had cases where the parallel runs do work once during a session, but then not terminating on a rerun.

Sometimes when closing down the session (after manually terminating the parallel run), I get the below message

```r
# Error while shutting down parallel: unable to terminate some child processes
```

Upon googling this, I found that I could do the following check.

```r
parallel::mcparallel(scan(n = 1, quiet = TRUE))
# $pid
# [1] 32763

# $fd
# [1] 21 24

# attr(,"class")
# [1] "parallelJob" "childProcess" "process"
```

Does anyone have any solutions to this? Is this perhaps a bug? Related to `future` or perhaps `parallel` or other package? I was having the same issue when I was running the `mlr` package.

```r
> Sys.info()
sysname
"Linux"
release
"4.15.0-66-generic"
version
"#75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019"
...
machine
"x86_64"
...
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.