Proposal: Envoy mTLS Private Key Protection with HSM

## Envoy mTLS Private Key Protection with HSM
Envoy uses boringssl for tls communication, and the enabling of mTLS can also protect the connection between services from being attacked and tampered, but there’s still vulnerabilities within the private key management in Envoy, as the private key is exposed as clear text data inside the memory. The proposal is to provide a selectable choice to enable the PKCS#11 standard inside Envoy together with Hardware Security Module (HSM) to serve better security for key management within Envoy.
### Design
This proposal tends to increase the security level of cryptographic operations and key management inside Envoy. It will modify Envoy to provide an advanced-level security to save the secrets (Istio-Agent mTLS private key) securely inside HSM (SoftHSM as sample) based on the PKCS#11 standard.
#### Control plane
The workflow of control plane is shown in the diagram below.
The main changes of workflow are shown by the blue arrows.
**1.2 1.3** The SDS request no longer requires control plane to generate key pair and CSR. And the SDS response sent to envoy does not directly contain the private key, but config to generate HSM private key provider. A kind of proto definition of SDS response is shown below. We define a stage field to distinguish two SDS responses, stage in the first SDS response is set to “init” to indicate the stage of creating the HSM private key provider,and stage in the second SDS response is set to “cert” to indicate that tls communication can be performed after receiving the certificate.
```
message Config {
string hsm_library = 1;
string key_label = 2;
string usr_pin = 3;
string so_pin = 4;
string token_label = 5;
string rsa_key_config = 6;
string ecdsa_key_config = 7;
string key_type = 8;
string stage = 9;
// csr_config is a string composed of various configuration items
// provided to envoy, which are used to generate CSR
string csr_config = 10;
}
```
**1.4 1.5 1.6** Then HSM private key provider will init HSM context once, and generate keypair in HSM and CSR with PKCS#11 engine.
**1.7** Since envoy’s SDS does not support CSR datatype resource, we add a new gRPC that reuse the original SDS channel, such as UDS. And CSR will be sent to control plane in this gRPC request.
**1.9 1.10** After control plane completes signing CSR, certificate will be sent to envoy through the second SDS response and HSM private key provider will be updated.
#### Data plane
The workflow of data plane is shown in the diagram below.
**2.3** When Boringssl perform TLS handshake in HTTPS connection, it will call signing/decryption function in HSM Private Key Provider.
**2.4 2.5 2.6** And HSM will execute signing or decryption and return the result to HSM Private Key Provider.
**2.7** Finally HSM Private Key Provider will return the result to Boringssl.

#### Key generation
When Envoy receives the first SDS response, the private key will be created in HSM when creating HSM key provider.And when Envoy receives the second SDS response, the private key will be loaded from HSM.
For both cases, the output returned is a key handle instead of the plain-text key.
#### Key retrieval
While following the PKCS#11 standard, id and label are used as the identifiers to fetch the private key. Either of them is usable to fetch the identical key. And the key that was retrieved is a handle instead of the plain-text data block.

We also raise a proposal in istio community: https://github.com/istio/istio/issues/36296.
@lizan @ggreenway @asraa @mattklein123