futureverse / futureverse/parallelly

ROBUSTNESS: Protect main-worker communication using suspendInterrupts()?

Open
#29 14 comments 1 reaction 0 assignees View on GitHub
Dominant language
R
Stars
140
Forks
9
PR merge metrics
No merged PRs in 30d

Description

# Issue

If the user hits Ctrl-C (signals a user interrupt) while the main R session and a worker communicates data, then the communication ends up in an unrecoverable corrupt. The only solution is to restart with a new cluster while waiting for the old cluster node to timeout (30 days?)

# Suggestion

In R (> 3.5.0), we have `suspendInterrupts(expr)` that suspends interrupts while evaluating an expression.

Could we wrap all communication calls, i.e. all `serialize()`/`unserialize()` calls in `suspendInterrupts()`?

There should be no need to do this on workers. Also, this way the worker can be terminated by the operating system or a job scheduler by signaling a nicer interrupt signal.

It should probably also be sufficient to protect interactive R sessions. When running R in batch mode, hitting Ctrl-C often means we want the whole R process to terminate. OTH, with proper interrupt handling (e.g. protecting communication as above and then capture user interrupts outside), our R process could terminate nicely, which here means calling `stopCluster()` etc.

# Actions

Investigate exactly which type of interrupt signals are suspended.

Protect what can be protected in the existing parallelly code.

Document that Ctrl-\ can be used to kill R if above get stuck. (What happens in RStudio?)

Contributor guide

Open the contributing guide

Research direction

No file or test is named. Start by locating the existing parallelly communication calls that use serialize() and unserialize(), then investigate which interrupts suspendInterrupts() covers; done means protecting the applicable communication paths and documenting Ctrl-\\ behavior, including the RStudio question.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.