futureverse / futureverse/parallelly
WISH: Shutting down clusters upon session exit
- Dominant language
- R
- Stars
- 140
- Forks
- 9
- PR merge metrics
- No merged PRs in 30d
Description
If the parent session of a PSOCK cluster terminates without properly shutting down the cluster first, the workers will (i) keep running, and (ii) eventually produce an error due to a broken connection:
```sh
$ R --vanilla
> cl <- parallelly::makeClusterPSOCK(1L, outfile='')
> quit('no')
starting worker pid=27586 on localhost:11470 at 20:15:04.478
Error in unserialize(node$con) : error reading from connection
Calls: workRSOCK ... doTryCatch -> recvData -> recvData.SOCKnode -> unserialize
$
```
# Idea
We could set up an "on-session-exit" function that will be called when R terminates. Here's a proof of concept:
```sh
$ R --quiet
> cl <- parallelly::makeClusterPSOCK(1L, outfile='')
> suppressMessages(library(R.utils)); onSessionExit(function() parallel::stopCluster(cl))
> quit('no')
```
The `onSessionExit()` function of **R.utils** works by injecting/updating a `.Last()` function, which will be called when calling `quit(..., runLast = TRUE)` which is the default. BTW, it looks like there's a [bug in **R.utils** causing this to only work if **R.utils** is attached - it's not sufficient to just loading](https://github.com/HenrikBengtsson/R.utils/issues/124), e.g. `R.utils::onSessionExit()` won't do.
Here is the same example when running in non-interactive mode;
```r
$ Rscript -e "cl <- parallelly::makeClusterPSOCK(1L, outfile='')"
starting worker pid=29883 on localhost:11511 at 20:28:48.438
Error in unserialize(node$con) : error reading from connection
Calls: workRSOCK ... doTryCatch -> recvData -> recvData.SOCKnode -> unserialize
```
and
```r
$ Rscript -e "cl <- parallelly::makeClusterPSOCK(1L, outfile=''); suppressMessages(library(R.utils)); onSessionExit(function() parallel::stopCluster(cl))"
starting worker pid=29738 on localhost:11208 at 20:28:40.744
```
Contributor guide
Assessment
This issue has not been assessed yet.