Include networks in node assignments, instead of embedding them in attachments
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 3.7k
- Forks
- 676
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 6
Description
Networks in swarmkit predate secrets and configs, and thus predate the concept of node assignments, which are how we deliver a secret or config to a node only once.
Instead, networks are delivered to a node as part of the Attachments of a task. Because many tasks attached to the same network may be scheduled on a node at the same time, this means that identical copies of a Network can be found on every node, as well as in the Raft store. See the attachment protocol buffer:
// NetworkAttachment specifies the network parameters of attachment to
// a single network by an object such as task or node.
message NetworkAttachment {
// Network state as a whole becomes part of the object so that
// it always is available for use in agents so that agents
// don't have any other dependency during execution.
Network network = 1;
// List of IPv4/IPv6 addresses that are assigned to the object
// as part of getting attached to this network.
repeated string addresses = 2;
// List of aliases by which a task is resolved in a network
repeated string aliases = 3;
// Map of all the driver attachment options for this network
map<string,string> driver_attachment_opts = 4;
}
Instead, we should specify in NetworkAttachment objects only the Network ID, and we should send down the whole Network object as a node assignment, like a Secret or Config.
The upgrade path for this is slightly more complicated than usual, because it would break backward compatibility of older agents to newer managers if used directly. Instead, the upgrade would have to take place over a few releases:
- In the first release, the managers will include both Networks as assignments and Networks included in the NetworkAttachment objects, sending both down to tasks. This allows forward and backward compatibility with agents. The worker would in turn be functional with at least any one of these two options present, allowing forward and backward compatibility.
- In the second release, managers stop including Networks in the NetworkAttachment objects, sending them down only as assignments. This would work with the worker from the previous version. The Agent will still accept both forms of network, but would prefer the Assignment and issue a warning if the network was only found in the attachment.
- Some time later, agents will stop looking for Networks in the task's attachments altogether, reducing the code complexity required to look for both.
This seems like a major API change, but will be relatively painless in practice because this change only alters the internal swarmkit API, which is not intended to be publicly exposed. The engine can easily maintain backward compatibility by In addition, the nature of protocol buffers is to be resilient to this kind of change. The Docker API would maintain backward compatibility because its external representation of the NetworkAttachment is a separate object, and we can simply re-embed the Network into the NetworkAttachment object when servicing requests at that level.
This same approach can be taken for Endpoints, which are also duplicated and embedded in every task.
This very nearly fix #2655, as well as improve the scalability of swarm.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the swarmkit task attachments, node assignments, manager, and worker paths described in the issue, then inspect the NetworkAttachment protocol buffer. Map the three-release compatibility plan, including the fallback warning behavior. Done means Networks are delivered as assignments while older agents remain functional, with later removal of attachment lookup and equivalent handling for Endpoints.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100