moby / moby/swarmkit

Having large numbers of networks can cause node allocation to fail

Open
#2,655 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

area/networking kind/bug
Dominant language
Go
Stars
3.7k
Forks
676
Avg merge
4d 9h
Merged PRs (30d)
6

Description

Since 9fa9ce1b5280f3f5d3ab659e3bd88cda0fb305a3, nodes are now allocated with a network attachment for every network.

Unfortunately, this can cause the size of a Node object to exceed the maximum raft message size, which prevents it from being committed to the object store.

From moby/moby#36792

Forgot to mention the same error was triggered by a different action than the one initially reported on this issue, in my case it appeared when creating ~1700 networks and deploying one service on each network.

Jun 01 00:39:52 ip-172-16-0-128 dockerd[1284]: time="2018-06-01T00:39:52.220386603Z" level=error msg="Failed to commit allocation of network resources for node rul9pnxcc2hpj3o7eya1redpk" error="raft: raft message is too large and can't be sent" module=node node.id=281crvu

There are many possible fixes, including disallowing too many attachments, but one way or another we can't wind up with raft messages too large like this.

/cc @ctelfer

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with commit 9fa9ce1b5280f3f5d3ab659e3bd88cda0fb305a3 and trace node allocation through the object store and Raft message handling. Reproduce the failure with many network attachments and inspect the reported maximum-message error. Done means node allocation cannot produce Raft messages that exceed the allowed size.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.