JuliaAI / JuliaAI/MLJTuning.jl
replace @distributed with pmap
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 70
- Forks
- 12
- PR merge metrics
- No merged PRs in 30d
Description
Currently in MLJ acceleration with CPUThreads is implemented using @distributed. This effectively splits up the given range (1:nfolds or 1:nmetamodels) into equal chunks and sends them off to all workers loaded with addprocs. This is great if the each chunk runs in the same amount of time otherwise some overhead is experienced. Also the user lacks the ability to specify the actual workers to be used in computing. (This might not be a big deal)
pmap implementation allows user more control (if they wish) in how these tasks are sent to to these workers.(this is due to batch_size and AbstractWorkerPool options it exposes).
Previously the main reason for not adopting pmap was because nested pmap hangs see https://github.com/JuliaLang/Distributed.jl/issues/62 (There is a workaround this stated there).
The only limitation left in adopting this is that calling pmap from within Threads.@spawn some times hangs.( Although i don't think it is practical to call pmap from threads. What is more common is calling threads from processes) see https://github.com/JuliaLang/Distributed.jl/issues/69
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the CPUThreads acceleration code that currently uses @distributed for 1:nfolds or 1:nmetamodels, then review Julia Distributed's pmap, batch_size, and AbstractWorkerPool behavior. Check the linked Distributed.jl issues for nested pmap and Threads.@spawn limitations; done means the acceleration uses pmap without introducing those hangs and retains the existing computations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- distributed-systems, machine-learning
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100