JuliaLang / JuliaLang/Distributed.jl
Setup worker-worker connections lazily
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 55
- Forks
- 19
- PR merge metrics
- No merged PRs in 30d
Description
The default all_to_all topology connects all processes to each other. While this is fine for small clusters, the total number of TCP connections increases rapidly as (N^2)/2.
Considering that a large class of parallel problems only need master-worker connections we should change the default topology to all_to_all_lazy where worker-worker connections are setup only on the first request from a worker to another worker. And also introduce another topology master_routed which only connects master to workers, and in case of a worker-worker call, routes the request through the master.
To summarize, implement 2 new topologies:
-
all_to_all_lazywhere worker-worker connections are setup lazily, and is the default for addprocs and -
master_routedin which only the master connects to workers and worker-worker messages are routed via the master.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the existing all_to_all topology implementation and the code that selects the default topology for addprocs. Trace how worker-worker connections and messages are established, then identify the tests covering topology behavior. Done means both requested topologies work, the new lazy topology is the addprocs default, and the stated routing and connection properties are verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- distributed-systems, networking
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 20/100