Internode TLS certificates dual used as both client and server certificates

Expected Behavior

• Ability to feed separate certificates for clients (eg: matching client, history client) and their respective server certificates for each service
• Ability to feed different server names for each component (Frontend, matching, history)

Actual Behavior

• Assumption in code to use single certificate for dual purpose. This puts a security risk with how mTLS is implemented. Based on discussions with temporal team it seems like self-signed certs were used during implementation of this feature causing this to fail.
• Another assumption in code to use same server name in SAN for all services (frontend, matching, history). This presents weak security as 1 microservice can impersonate another.

Steps to Reproduce the Problem

1. Create a server certificate with the following in the certificate:
   X509v3 Extended Key Usage:
   TLS Web Server Authentication
2. Use the cert/key pair in the TLS/internode section and bring up FE, Matching, History, Worker service
3. Use tctl to query get-search-attributes

       Error: Unable to get search attributes.
Error Details: rpc error: code = Unavailable desc = unable to get temporal-sys-add-search-attributes-workflow workflow state: context deadline exceeded
('export TEMPORAL_CLI_SHOW_STACKS=1' to see stack traces)

If the above steps are repeated with a client certificate containing the following:
X509v3 Extended Key Usage:
TLS Web Client Authentication
Error observed:
{"level":"error","ts":"2022-04-01T01:15:45.045Z","msg":"matching client encountered error","service":"history","error":"last connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate specifies an incompatible key usage\"","service-error-type":"serviceerror.Unavailable","logging-call-at":"metricClient.go:259","stacktrace":"..."}

Specifications

• Version: 1.15.0
• Platform: Linux

Start with the TLS/internode configuration and the matching client error reported at metricClient.go:259. Reproduce the tctl get-search-attributes failure using certificates with server-only and client-only Extended Key Usage, then trace how Frontend, matching, and history select certificates and server names. Done means separate client/server certificates and per-component server names work without the x509 key-usage error.