Having large numbers of networks can cause node allocation to fail
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 3.7k
- Forks
- 676
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 6
Description
Since 9fa9ce1b5280f3f5d3ab659e3bd88cda0fb305a3, nodes are now allocated with a network attachment for every network.
Unfortunately, this can cause the size of a Node object to exceed the maximum raft message size, which prevents it from being committed to the object store.
From moby/moby#36792
Forgot to mention the same error was triggered by a different action than the one initially reported on this issue, in my case it appeared when creating ~1700 networks and deploying one service on each network.
Jun 01 00:39:52 ip-172-16-0-128 dockerd[1284]: time="2018-06-01T00:39:52.220386603Z" level=error msg="Failed to commit allocation of network resources for node rul9pnxcc2hpj3o7eya1redpk" error="raft: raft message is too large and can't be sent" module=node node.id=281crvu
There are many possible fixes, including disallowing too many attachments, but one way or another we can't wind up with raft messages too large like this.
/cc @ctelfer
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with commit 9fa9ce1b5280f3f5d3ab659e3bd88cda0fb305a3 and trace node allocation through the object store and Raft message handling. Reproduce the failure with many network attachments and inspect the reported maximum-message error. Done means node allocation cannot produce Raft messages that exceed the allowed size.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100