futureverse / futureverse/future

Share parallel::makeCluster over nested loop

Open
#208 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
R
Stars
1k
Forks
92
PR merge metrics
No merged PRs in 30d

Description

Hi HenrikBengtsson,

I am discovering the future package, and I was wondering if I could make it work for my particular use case.

In my case, I need to do large set of nested computations. Let's suppose that my case requires two layers of nesting. Both the inner and the outer layer take a long time to compute, and the time to execute both the inner and outer layers are wildly varying. This makes load balancing when I parallelize by 'traditional one-layer' methods particularly challenging. If I make the outer loop parallel, then I am often waiting waiting a long time for the last iteration to finish because the inner layers take a long time to compute. If I make only the inner loop parallel, then my outer loop is sometimes waiting long for one single inner task to finish before it can initiate the next iteration of outer calculations.

Given this use case, my code would benefit significantly from a futures approach with a local cluster using parallel::makeCluster(). This way, my function can just dive into double loop (depth first) and create some futures. At the end I will wait for all the futures to finish. In this way, all the workers/cores will always be used optimally and none of them will ever be idle up till the very end. Note however that I only have one computer available with multiple cores. So the cluster must be shared between the inner and outer loop.

Perhaps I am over-complicating things by attempting nested futures. It could very well be that `plan(list(sequential, tweak(cluster, workers=cl)))` already works good enough. My concern however is that the outer layer being sequential also means that the inner layer blocks the sequential progression of the outer layer until all futures are processed. I was not able to successfully test this myself.

Code speaks much louder than words. So I tried to make a simplified representation of what I want to achieve. Clearly, it is not working since passing the same cluster to the nested plan raises errors. However I think my intent from this simple example is clear. Hopefully you have any suggestions as to how to solve my case most efficiently. Your help will be greatly appreciated!

My example code (which currently does not work):
```
cl = makeCluster(4)

plan(list(tweak(cluster, workers = cl), tweak(cluster, workers = cl)))

x <- vector(mode = "list", length = 4)
for (i in 1:4) {
x[[i]] <- future({
tmp = slow_function_first_pass(i)
y <- vector(mode = "list", length = 3)
for (j in 1:3) {
y[[j]] <- future({slow_fuction_second_pass(tmp)})
}
y
})
}
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the nested-futures example with parallel::makeCluster() and the shared cluster described in the issue. Investigate how nested plans handle the same cluster and determine the intended behavior for sharing workers between outer and inner loops. Done should mean that the requested nested computation works without the reported errors, with its supported usage documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
distributed-systems, hpc
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.