JuliaAI / JuliaAI/MLJTuning.jl

replace @distributed with pmap

Open
#54 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
70
Forks
12
PR merge metrics
No merged PRs in 30d

Description

Currently in MLJ acceleration with CPUThreads is implemented using @distributed. This effectively splits up the given range (1:nfolds or 1:nmetamodels) into equal chunks and sends them off to all workers loaded with addprocs. This is great if the each chunk runs in the same amount of time otherwise some overhead is experienced. Also the user lacks the ability to specify the actual workers to be used in computing. (This might not be a big deal)
pmap implementation allows user more control (if they wish) in how these tasks are sent to to these workers.(this is due to batch_size and AbstractWorkerPool options it exposes).
Previously the main reason for not adopting pmap was because nested pmap hangs see https://github.com/JuliaLang/Distributed.jl/issues/62 (There is a workaround this stated there).
The only limitation left in adopting this is that calling pmap from within Threads.@spawn some times hangs.( Although i don't think it is practical to call pmap from threads. What is more common is calling threads from processes) see https://github.com/JuliaLang/Distributed.jl/issues/69

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the CPUThreads acceleration code that currently uses @distributed for 1:nfolds or 1:nmetamodels, then review Julia Distributed's pmap, batch_size, and AbstractWorkerPool behavior. Check the linked Distributed.jl issues for nested pmap and Threads.@spawn limitations; done means the acceleration uses pmap without introducing those hangs and retains the existing computations.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
distributed-systems, machine-learning
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.