JuliaLang / JuliaLang/Distributed.jl

Setup worker-worker connections lazily

Open
#42 7 comments 7 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
55
Forks
19
PR merge metrics
No merged PRs in 30d

Description

The default all_to_all topology connects all processes to each other. While this is fine for small clusters, the total number of TCP connections increases rapidly as (N^2)/2.

Considering that a large class of parallel problems only need master-worker connections we should change the default topology to all_to_all_lazy where worker-worker connections are setup only on the first request from a worker to another worker. And also introduce another topology master_routed which only connects master to workers, and in case of a worker-worker call, routes the request through the master.

To summarize, implement 2 new topologies:

  1. all_to_all_lazy where worker-worker connections are setup lazily, and is the default for addprocs and

  2. master_routed in which only the master connects to workers and worker-worker messages are routed via the master.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the existing all_to_all topology implementation and the code that selects the default topology for addprocs. Trace how worker-worker connections and messages are established, then identify the tests covering topology behavior. Done means both requested topologies work, the new lazy topology is the addprocs default, and the stated routing and connection properties are verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
distributed-systems, networking
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.