moby / moby/libnetwork

Time to create an overlay netowrk significantly increased as the number of agents increases.

Open
#1,672 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
2.2k
Forks
875
PR merge metrics
No merged PRs in 30d

Description

Hi, I compare the time creating an overlay network in a libnetwork cluster with different number of dnet.
In a 50-dnets setup, it take ~ 1s, which looks reasonable.

# time  ./bin/dnet -H tcp://127.0.0.1:2385 network create -d overlay multihost_50
45ed34c232aa40194a15a0da64432fc25b403780b6f1ea9bfee8ec5a650a174e

real	0m1.540s
user	0m0.012s
sys	0m0.008s

However, in a 100-dent cluster it takes about 10s

# time  ./bin/dnet -H tcp://127.0.0.1:2385 network create -d overlay multihost_100
3e28785cd3d5fd1b8cfd8574b8511053e356216a101652a7e0fe641aff5cc847

real	0m10.098s
user	0m0.004s
sys	0m0.012s

The idle CPU is 90%, so the system is not saturated.

In my understanding, the network resource has global scope.
So besides watching for the KV, does the overlay driver do something else with the network KV which slows down this operation? Thanks.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the reported ./bin/dnet -H tcp://127.0.0.1:2385 network create -d overlay timings with 50- and 100-dnet clusters. Start by tracing what the overlay driver does with the network KV beyond watching it; done means identifying the source of the scaling slowdown and verifying the explanation against both measurements.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.