futureverse / futureverse/parallelly

WISH: Shutting down clusters upon session exit

Open
#31 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
R
Stars
140
Forks
9
PR merge metrics
No merged PRs in 30d

Description

If the parent session of a PSOCK cluster terminates without properly shutting down the cluster first, the workers will (i) keep running, and (ii) eventually produce an error due to a broken connection:

```sh
$ R --vanilla
> cl <- parallelly::makeClusterPSOCK(1L, outfile='')
> quit('no')
starting worker pid=27586 on localhost:11470 at 20:15:04.478
Error in unserialize(node$con) : error reading from connection
Calls: workRSOCK ... doTryCatch -> recvData -> recvData.SOCKnode -> unserialize
$
```

# Idea

We could set up an "on-session-exit" function that will be called when R terminates. Here's a proof of concept:

```sh
$ R --quiet
> cl <- parallelly::makeClusterPSOCK(1L, outfile='')
> suppressMessages(library(R.utils)); onSessionExit(function() parallel::stopCluster(cl))
> quit('no')
```

The `onSessionExit()` function of **R.utils** works by injecting/updating a `.Last()` function, which will be called when calling `quit(..., runLast = TRUE)` which is the default. BTW, it looks like there's a [bug in **R.utils** causing this to only work if **R.utils** is attached - it's not sufficient to just loading](https://github.com/HenrikBengtsson/R.utils/issues/124), e.g. `R.utils::onSessionExit()` won't do.

Here is the same example when running in non-interactive mode;

```r
$ Rscript -e "cl <- parallelly::makeClusterPSOCK(1L, outfile='')"
starting worker pid=29883 on localhost:11511 at 20:28:48.438
Error in unserialize(node$con) : error reading from connection
Calls: workRSOCK ... doTryCatch -> recvData -> recvData.SOCKnode -> unserialize
```

and

```r
$ Rscript -e "cl <- parallelly::makeClusterPSOCK(1L, outfile=''); suppressMessages(library(R.utils)); onSessionExit(function() parallel::stopCluster(cl))"
starting worker pid=29738 on localhost:11208 at 20:28:40.744
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.