futureverse / futureverse/progressr

Utility functions for relaying stdout and conditions when using mclapply()/parLapply(), ...

Open
#51 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
R
Stars
299
Forks
11
PR merge metrics
No merged PRs in 30d

Description

I'm adding this one to this package, but it probably belongs to a 'parallel.extras' package:

Add some type of utility functions for relaying stdout and conditions when using `mclapply()`, `parLapply()`, ... of the 'parallel' package. Not sure what such an API would look like, but it might help us support 'progressr' also in those cases, because current the following will relay nothing from the cluster nodes:
```r
library(progressr)
options(progressr.interval = 0.0, progressr.delay = 0.01)

cl <- parallel::makeCluster(3)
with_progress({
p <- progressor(10)
parallel::clusterExport(cl, c("p", "slow_sum"))
y <- parallel::parLapply(cl, X = 1:10, function(x) {
p()
slow_sum(x, stdout=TRUE, message=TRUE)
})
})
parallel::stopCluster(cl)
```

Maybe something that will allow us to do:
```r
res <- parallel::parLapply(cl, X = 1:10, function(x) record_output({
p()
slow_sum(x, stdout=TRUE, message=TRUE)
}))
y <- value(res) ## cf. how futures work
```

Contributor guide

Open the contributing guide

Research direction

Start with the mclapply() and parLapply() examples in the issue, then read the parallel package behavior for cluster workers and compare it with the progressr/futures model mentioned. Done would require a settled API for relaying stdout, conditions, and progress updates, plus documented behavior for retrieving the resulting values.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.