moby / moby/swarmkit

Include networks in node assignments, instead of embedding them in attachments

Open
#2,658 2 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/networking exp/expert kind/enhancement kind/proposal
Dominant language
Go
Stars
3.7k
Forks
676
Avg merge
4d 9h
Merged PRs (30d)
6

Description

Networks in swarmkit predate secrets and configs, and thus predate the concept of node assignments, which are how we deliver a secret or config to a node only once.

Instead, networks are delivered to a node as part of the Attachments of a task. Because many tasks attached to the same network may be scheduled on a node at the same time, this means that identical copies of a Network can be found on every node, as well as in the Raft store. See the attachment protocol buffer:

// NetworkAttachment specifies the network parameters of attachment to
// a single network by an object such as task or node.
message NetworkAttachment {
	// Network state as a whole becomes part of the object so that
	// it always is available for use in agents so that agents
	// don't have any other dependency during execution.
	Network network = 1;

	// List of IPv4/IPv6 addresses that are assigned to the object
	// as part of getting attached to this network.
	repeated string addresses = 2;

	// List of aliases by which a task is resolved in a network
	repeated string aliases = 3;

	// Map of all the driver attachment options for this network
	map<string,string> driver_attachment_opts = 4;
}

Instead, we should specify in NetworkAttachment objects only the Network ID, and we should send down the whole Network object as a node assignment, like a Secret or Config.

The upgrade path for this is slightly more complicated than usual, because it would break backward compatibility of older agents to newer managers if used directly. Instead, the upgrade would have to take place over a few releases:

  1. In the first release, the managers will include both Networks as assignments and Networks included in the NetworkAttachment objects, sending both down to tasks. This allows forward and backward compatibility with agents. The worker would in turn be functional with at least any one of these two options present, allowing forward and backward compatibility.
  2. In the second release, managers stop including Networks in the NetworkAttachment objects, sending them down only as assignments. This would work with the worker from the previous version. The Agent will still accept both forms of network, but would prefer the Assignment and issue a warning if the network was only found in the attachment.
  3. Some time later, agents will stop looking for Networks in the task's attachments altogether, reducing the code complexity required to look for both.

This seems like a major API change, but will be relatively painless in practice because this change only alters the internal swarmkit API, which is not intended to be publicly exposed. The engine can easily maintain backward compatibility by In addition, the nature of protocol buffers is to be resilient to this kind of change. The Docker API would maintain backward compatibility because its external representation of the NetworkAttachment is a separate object, and we can simply re-embed the Network into the NetworkAttachment object when servicing requests at that level.

This same approach can be taken for Endpoints, which are also duplicated and embedded in every task.

This very nearly fix #2655, as well as improve the scalability of swarm.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the swarmkit task attachments, node assignments, manager, and worker paths described in the issue, then inspect the NetworkAttachment protocol buffer. Map the three-release compatibility plan, including the fallback warning behavior. Done means Networks are delivered as assignments while older agents remain functional, with later removal of attachment lookup and equivalent handling for Endpoints.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.