Service Identity feature

Background

Mutually authenticated TLS (MTLS) is the gold-standard for authentication, authorization and encryption between microservices. It not only provides a battle tested, granular way of doing application segmentation, but also provides strong cryptographic guarantees to all the traffic that flows between applications. Unfortunately, in order to be able to fully roll-out MTLS in any companies’ infrastructure, there are several problems that have to be solved first, such as having a consistent application-naming scheme, having the ability of issuing and rotating certificates from multiple PKI’s and having a secure secret distribution mechanism in place. The objective of the Service Identity project in docker is to allow ops teams to easily deploy MTLS across all of their services, providing a mechanism that allows any services deployed in a swarm to authenticate, authorize and encrypt communications between each other.

What our solution provides

The ability to easily create and manage distinct service-level PKIs in a swarm.
Automatic issuance of TLS certificates for running services in desired PKIs.
Automatic rotation of TLS certificates that will expire using swarm’s rolling deploys
The ability to manage the roots of trust for each service.
A stable service ID scheme for services deployed through swarm that can be used for service-to-service authentication.
Out of the box integration with default TLS servers and clients
A SPIFFE Verifiable Identity Document to use with SPIFFE-capable services

What our solution will not provide

An ACL for granular service-to-service authorization.
A higher-level framework that allows for complex authorization around which particular endpoints a remote peer is allowed to call.

Design doc

https://docs.google.com/document/d/117lKj_VxYa2UvVhx_md51ldjziU_1J1gYKJlGc1Ncrg/edit#heading=h.bpoa5txf40k1

TODO (WIP)

• Update secret object for certificates, that has a public component, private component, arbitrary external CAs, and expiry - can possibly make use of the api.RootCA object, which already handles rotation, and the api.ExternalCA object. (https://github.com/docker/swarmkit/pull/2331)
• Allow certificate secrets to reference other certificate secrets - possibly we should also add a flag to indicate that a secret is generative, and thus completely immutable by users. We need the generated certificates to reference back to the root that issued them, so we can determine which certificates/services that need changing/redeploying when a root CA rotates. The backreference is also needed to bundle the trust roots when the generated certificate/key are shipped. (https://github.com/docker/swarmkit/pull/2331)
• Generalize ca.rootRotationReconciler object to take interfaces and not be as intricately tied to the cluster root CA (e.g. it can make use of node health checks to determine if a node is up, and the check w.r.t. to the node TLSInfo can be moved into something with a similar interface to the health check). The idea is to have the swarm leader run one reconciler per PKI that’s in the middle of rotating.
• Create a component that runs on the leader that monitors leaf certificate expiries and hooks that up to the TLS renewer. It should trigger redeploys of linked services whenever a leaf certificate is renewed.
• Create a component that runs on the leader that monitors for any root certificate rotations, and creates or updates a root reconciler for each one.
• Create control API for creating PKI, and attaching services to PKI
• Integration into moby/moby etc.

Testing (WIP)

• When a service is created that references a PKI, a new service identity is generated for that service. When the service is removed, the service identity is removed. Two services referencing the same PKI should not have the same service identity.
• When leaf cert across multiple service are nearing expiry, tasks using a near-expiry cert are restarted by default (bumps version, so ignores restart policy).
• When a root cert nears expiry, the root certificate, if generated by swarm, is automatically renewed and all services using leaf certs restarted.
• When a root cert is rotated, the leaf certificates are rotated and all services restarted.
• In the swarm control API, the private keys for PKIs are redacted. The spec’s desired cert and key are redacted.

Start with the linked design document and the existing api.RootCA, api.ExternalCA, and ca.rootRotationReconciler work referenced in the TODOs. Review the swarm control API and current certificate and secret tests; no specific test file is named. Done means addressing the listed PKI, certificate lifecycle, service identity, API, integration, rotation, and redaction requirements with the proposed tests passing.