mTLS cert expiry notification and upgrade needs to be improved for production clusters to prevent outages
- Dominant language
- Go
- Stars
- 353
- Forks
- 221
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 2
Description
## Issue
Currently Dapr makes is very easy to create a root and issuing certs when the control plane is first installed into a hosting environment, such as cluster (for example kubernetes). The sentry service is used to generate and distribute certs to the side-cars and provide mTLS capabilities, which are used for service invocation. This is an important Dapr capability for operators. See https://docs.dapr.io/operations/security/mtls/
However the root cert generated by the Sentry service has a **1 year expiry date** from generation and this means that, once a cluster has been running in production for a year the certs need to be updated. As described in the above article this is left as a manual process currently, with the operator needing to generate new certs and manually copy these into a K8s secret. This is hard, awkward and error prone. In order to prevent issues in production cluster this process of cert upgrades needs to be improved. Given the v1.0 anniversary of Dapr v1.0 is Feb 2022, we would start to see many production clusters that have been running for 1 year start to hit this issue. Generally in distributed applications certificate expiry is one of the most common issues to affect long running clusters and causes incidents to be reported and application downtime. This needs to be prevented.
## Describe the proposal
Improving the process for certification upgrade can be achieved with the following;
1) Have a notification event/metric event from the Dapr Sentry service raised to the operator every hour once the root expiry certificate is less than 30 days away from expiration. Currently the expiry date for the cert can be found with
`dapr mtls expiry -k`
but this is a manual process and automated warnings should also be surfaced in logs.
2) The mtls command needs to have `upgrade` and `import` options to ensure successful certs rollovers.
Ideally the operator should be able to run the following CLI command to simply upgrade the root and issue certs in the cluster with a new expiry date (say another 1 or 2 years from the current date)
`dapr mtls upgrade --kubernetes --expirydate `
The goal of this command is to simplified the upgrade process for soon to be expiring CA and Issuer certs for an operator and prevent application downtime.
This command, generates a new root and issuer issue certificate with the specified future date alongside the existing root and issuer certificate. Ideally a push based approach would push the new cert to each of the dapr sidecars, however instead a rolling upgrade of the dapr deployments may be needed by the operator to pick up the new cert. The old cert can be left in place and cleaned up next time the upgrade command is run, so that there are only ever two (previous and current) certs installed into the environment. Alternatively when updating the root certificate if it uses the _same_ private key then the Dapr instance would not then need to be restarted. This may be a configuration option to consider.
3) For those wanting to provide their own certs, both initially and during upgrade, the current manual process of editing the K8s secrets (or any other future hosting environment) is open to mistakes. For example having to use Helm as another tool. Having a specific CLI import command to install the certificates correctly hides this complexity and improves certification update.
Ideally the operator should be able to run the following CLI command
`dapr mtls import --kubernetes --certificateAuthorityRootCertificate --issuerPrivateKey --issuerPublicCertificate`
(note: This should be used instead of the K8s Helm command, in the same way that `dapr init` hides the use of Helm for dapr installation as the recommended way to install dapr )
4) Make it easy to generate new certs with the correct format that can be imported, by wrapping the openssl command with a CLI command. This is particularly useful for certs being used for local self-host testing.
`dapr mtls createcertificates --path `
The output of this command can be used by the `dapr mtls import -k` command
##Optional. Improve the local mTLS usage experience.
5) Using the Sentry service on a local installation could also be improved with the above CLI commands. Ideally there could also be a CLI command to start the Sentry service locally using generated certificates placed into a known folder. In other words improving the multiple steps here. https://docs.dapr.io/operations/security/mtls/#self-hosted, although this is not as important as the upgrade command above. Something like this;
`dapr init --enable-mtls` - also starts and configures the Sentry service locally (today this does nothing) and generates certs using the `dapr mtls createcertificates` command into a known folder
`dapr mtls` can be now run without having to use the -k option to see the Sentry service running local and the mtls is enabled in self hosted mode
Also see this related issue on the local Sentry service https://github.com/dapr/cli/issues/550
Contributor guide
Assessment
This issue has not been assessed yet.