futureverse / futureverse/progressr
Utility functions for relaying stdout and conditions when using mclapply()/parLapply(), ...
- Dominant language
- R
- Stars
- 299
- Forks
- 11
- PR merge metrics
- No merged PRs in 30d
Description
I'm adding this one to this package, but it probably belongs to a 'parallel.extras' package:
Add some type of utility functions for relaying stdout and conditions when using `mclapply()`, `parLapply()`, ... of the 'parallel' package. Not sure what such an API would look like, but it might help us support 'progressr' also in those cases, because current the following will relay nothing from the cluster nodes:
```r
library(progressr)
options(progressr.interval = 0.0, progressr.delay = 0.01)
cl <- parallel::makeCluster(3)
with_progress({
p <- progressor(10)
parallel::clusterExport(cl, c("p", "slow_sum"))
y <- parallel::parLapply(cl, X = 1:10, function(x) {
p()
slow_sum(x, stdout=TRUE, message=TRUE)
})
})
parallel::stopCluster(cl)
```
Maybe something that will allow us to do:
```r
res <- parallel::parLapply(cl, X = 1:10, function(x) record_output({
p()
slow_sum(x, stdout=TRUE, message=TRUE)
}))
y <- value(res) ## cf. how futures work
```
Contributor guide
Research direction
Start with the mclapply() and parLapply() examples in the issue, then read the parallel package behavior for cluster workers and compare it with the progressr/futures model mentioned. Done would require a settled API for relaying stdout, conditions, and progress updates, plus documented behavior for retrieving the resulting values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100