futureverse / futureverse/future

worker "pool" for nested paralellization

Open
#361 2 comments 0 reactions 0 assignees View on GitHub
Backend API feature request
Dominant language
R
Stars
1k
Forks
92
PR merge metrics
No merged PRs in 30d

Description

If I understand correctly, `plan(tweak(multicore, workers=8))` means that the first nesting level gets 8 parallel threads and the second nesting level gets no parallelism. I could hard-allocate threads to each level, but that's hard to do since it means I have to know all thread usages down the tree of packages.

What I'm looking for is a "worker pool" like implementation. A naive greedy allocation using a semaphore that decrements every time a thread is forked off would be a good start. So that if I have a loop of three calling a package that has uses `future.apply` on a huge vector but takes very long to even get there, the NN workers can be busy for as much of the time as possible.

Interaction with in particular OMP is a problem of course. A lot of things seem to use that. IRC, Intel TBB auto-detects the number of "useful" threads to use and adjusts this value as it goes based on system load. Something like this would need extra house keeping, but the concept of "don't start more threads if all my workers/cpus are busy", or even "don't start more threads if we are at XY% memory" would be very useful to robustly run things in parallel.

Contributor guide

Open the contributing guide

Research direction

Start by examining how plan(tweak(multicore, workers=8)) handles nested parallelization and how future.apply uses workers. Investigate the requested semaphore-style allocation alongside OMP and Intel TBB behavior; done would mean nested work can share a worker pool without exceeding available workers.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
distributed-systems, hpc
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.