futureverse / futureverse/future

Need API like parallel::clusterCall

Open
#273 2 comments 6 reactions 0 assignees View on GitHub
Backend API feature request feature/sticky-globals globals
Dominant language
R
Stars
1k
Forks
92
PR merge metrics
No merged PRs in 30d

Description

I need to do some setup on each node in a multisession backend before creating any future. Currently, there's no API like `parallel::clusterCall` or `parallel::clusterEvalQ` to call function or evaluation an expression on each node of a cluster and `multisession` does not seem to expose to user the PSOCK cluster it creates.

Therefore I have to create my own cluster with

```r
library(future)
library(parallel)
cl <- makeClusterPSOCK(30)
clusterCall(cl, function() {
# setup code
})
plan(cluster, workers = cl)
```

It would be very nice to provide API on cluster level to make it easier to perform backend setup. If the setup code relies on some variables in calling scope, it would require user to call `parallel::clusterExport` to export all used variables before `parallel::clusterCall`, which is best handled by `future` package just like how variables are resolved when calling `future`.

This issue looks similar to #272.

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing multisession and cluster entry points, then compare their behavior with parallel::clusterCall, parallel::clusterEvalQ, and parallel::clusterExport. The requested API should allow setup or expression evaluation on multisession workers before futures are created, including handling variables from the calling scope; verify behavior with a focused cluster setup test.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.