Consul Connect’s auto_encrypt CA violates RFC5280 4.1.2.6 (creates a client agent certificate with no issuer in secondary DCs)

#### Overview of the Issue

We have `auto_encrypt` (using the Consul Connect CA) enabled for our Consul cluster, and we generate our own server certificates. We have consul clients deployed to Kubernetes clusters using the upstream Consul Helm chart (with `client.hostNetwork: true`). The client certificates work fine when we issue a HTTPS request from a Kubernetes pod to the Consul client agent that is joined to the primary DC.

When issuing HTTPS requests from a pod to a Consul client agent that is joined to a _**secondary DC**_, we face an `SSL error (60) X509-issuer not found`. When we inspect the certificate chain of the Consul client in the secondary DC, we find the Consul client certificate to have an empty issuer.

For reference, we supplied the Consul client in the primary DC with the Consul Connect root CA that was fetched from the primary DC servers (using `curl -s -k {IP}:{PORT}/v1/connect/ca/roots`). And we were able to make requests such as `curl --cacert client-cert.pem -H "X-Consul-Token":"XX" https://localhost:8501/v1/catalog/services` after.

The same doesn’t work in a secondary DC. We would fetch the certificate from the secondary DC, but would get two: a root and an intermediate certificate. Both have a valid issuer, but the intermediate has an empty subject. Since the subject is empty, the Consul client certificate has an empty issuer. So, when we try to make an HTTPS request with any of these certs to the Consul client agent, we get the error above.

As per RFC5280, the Issuer must not be empty and that if the certificate belongs to a CA, then the Subject also cannot be empty. (And, in the same sentence, that the CA must copy its own Subject to the Issuer to all certificates it issues).

Consul's behavior is compliant with [section 4.2.1.6]([https://www.rfc-editor.org/rfc/rfc5280#section-4.2.1.6](https://www.rfc-editor.org/rfc/rfc5280#section-4.1.2.6)) of RFC5280, but is violating [section 4.1.2.6](https://www.rfc-editor.org/rfc/rfc5280#section-4.1.2.6), which is causing issues in clients that do not allow certificates with empty issuer fields.

This behavior may have been introduced in this PR: https://github.com/hashicorp/consul/pull/10424

These clients currently do not work with TLS verification on:
- cURL
- Consul template 0.18 or older
- Some Java springboot clients (even if TLS verify is turned off)

PR fix for this issue (fork):
* https://github.com/criteo-forks/consul/commit/c9e2fb210df9a696c3ae481d7a887f6b54b0df2b
* https://github.com/hashicorp/consul/pull/10424

#### Reproduction Steps

1. Create a cluster with 2 DCs, one primary and one secondary (with at least 1 client agent each)
1. Enable `auto_encrypt`
1. Create server certificates (https://developer.hashicorp.com/consul/tutorials/security/tls-encryption-secure#create-the-server-certificates)
1. Distribute client certificates using `auto-encrypt` (https://developer.hashicorp.com/consul/tutorials/security/tls-encryption-secure#client-certificate-distribution)
1. Inspect the certificate on the Consul client connected to the secondary DC
- `openssl s_client -connect localhost:8501 -showcerts`
- The client agent certificate will have no issuer (see example certificates below)

1. Use the CA certificate from the server in the secondary DC to get the Connect root CA certificate
- `curl -s --cacert :8501/v1/connect/ca/roots | jq`
- Use the intermediate certificate to initiate a curl request on the client
```
curl --cacert client-cert.pem https://localhost:8501/v1/catalog/services

curl: (60) SSL: couldn't get X509-issuer name!
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
```

Note: this error happens because some clients (like cURL, Java springboot and some older Consul template versions <0.18) do not accept any certificates that have an empty issuer field, as that is a violation of section [section 4.1.2.6](https://www.rfc-editor.org/rfc/rfc5280#section-4.1.2.6) of the RFC.

Consul Connect CA root certificate

```bash
/ # curl localhost:8500/v1/connect/ca/roots | jq '.Roots[0].RootCert'

"-----BEGIN CERTIFICATE-----\nMIICDjCCAbWgAwIBAgIBBzAKBggqhkjOPQQDAjAxMS8wLQYDVQQDEyZwcmktNzU5\nMXUzOXAuY29uc3VsLmNhLjQzNzc1OTQ2LmNvbnN1bDAeFw0yMzAyMDgxNDUwMzJa\nFw0zMzAyMDUxNDUwMzJaMDExLzAtBgNVBAMTJnByaS03NTkxdTM5cC5jb25zdWwu\nY2EuNDM3NzU5NDYuY29uc3VsMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEeuOD\nyJmhmAL/UUELBWMYk/60AX3myXlxd6iqhX0B8qpKALfnvAchyYIWgM24fKFyxZtu\nEj5HvVGyNopZhbSKlaOBvTCBujAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUw\nAwEB/zApBgNVHQ4EIgQgPFwgaragEzCwsA3p4geTd0PyEfmS8mhHYlKAdbhG6nMw\nKwYDVR0jBCQwIoAgPFwgaragEzCwsA3p4geTd0PyEfmS8mhHYlKAdbhG6nMwPwYD\nVR0RBDgwNoY0c3BpZmZlOi8vNDM3NzU5NDYtMGM3Ni01YmRiLTg3MDItYzkwYThj\nNTI3ZGI1LmNvbnN1bDAKBggqhkjOPQQDAgNHADBEAiBfmuf1bMS+evFH2GZzUx7q\nDF1FUE7bjbphu7NMNajIUAIgeig+lq2IrUEEYTDbsbAE1KHS820mz2jKdBGn+56v\naRY=\n-----END CERTIFICATE-----\n"

```

Consul Connect CA intermediate certificate (secondary DC)

```
/ # curl localhost:8500/v1/connect/ca/roots | jq '.Roots[0].IntermediateCerts'
[
"-----BEGIN CERTIFICATE-----\nMIIB5DCCAYqgAwIBAgIBCjAKBggqhkjOPQQDAjAxMS8wLQYDVQQDEyZwcmktNzU5\nMXUzOXAuY29uc3VsLmNhLjQzNzc1OTQ2LmNvbnN1bDAeFw0yMzAyMDgxNDQ5MzRa\nFw0yNDAyMDgxNDQ5MzRaMAAwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAASYy150\nhIvw7ESsmEGssCq1ErR15uxuLeFQkzUiZhqZb364MYkOh0lglavOodlUSqiiNef0\npyk74S23/w6h/GZ3o4HDMIHAMA4GA1UdDwEB/wQEAwIBhjASBgNVHRMBAf8ECDAG\nAQH/AgEAMCkGA1UdDgQiBCDy4oluTlcIB+o+MmKiHcsZe/OxaEZEyihgZ+vz4W6t\nWDArBgNVHSMEJDAigCA8XCBqtqATMLCwDeniB5N3Q/IR+ZLyaEdiUoB1uEbqczBC\nBgNVHREBAf8EODA2hjRzcGlmZmU6Ly80Mzc3NTk0Ni0wYzc2LTViZGItODcwMi1j\nOTBhOGM1MjdkYjUuY29uc3VsMAoGCCqGSM49BAMCA0gAMEUCIQDN5JonU1Abl0P3\nyjvsq4Qqah6SVDioxsY03Qgejtv8DAIgdRNyUwR6SHyhgkD3uFgQ6wVvMXevg9WD\n/e8aHjFxV+8=\n-----END CERTIFICATE-----\n"
]
```

Consul client agent certificate with empty issuer (secondary DC)

```
/ # openssl s_client -connect localhost:8501 -showcerts
CONNECTED(00000003)
Can't use SSL_get_servername
depth=1
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0
verify return:1
---
Certificate chain
0 s:
i:
-----BEGIN CERTIFICATE-----
MIICFTCCAbugAwIBAgIBCzAKBggqhkjOPQQDAjAAMB4XDTIzMDIwODE0NDkzNFoX
DTIzMDIxMTE0NDkzNFowADBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABBo4ihSf
RnEruHsVxe0wvhBk+fHaVjJCm3imAM4vwgvVpcJdIYt4mUFjjKiz6MN/HvnKLIce
KarHouLz4eWP8XejggEkMIIBIDAOBgNVHQ8BAf8EBAMCA7gwHQYDVR0lBBYwFAYI
KwYBBQUHAwIGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwKQYDVR0OBCIEIOkmjW95
Lg8k12Jt8cyS/AxgVPv+c+rCunuQTrbq6TUyMCsGA1UdIwQkMCKAIPLiiW5OVwgH
6j4yYqIdyxl787FoRkTKKGBn6/Phbq1YMIGIBgNVHREBAf8EfjB8gglsb2NhbGhv
c3SHBH8AAAGHEAAAAAAAAAAAAAAAAAAAAAGGV3NwaWZmZTovLzQzNzc1OTQ2LTBj
NzYtNWJkYi04NzAyLWM5MGE4YzUyN2RiNS5jb25zdWwvYWdlbnQvY2xpZW50L2Rj
L2RjMi9pZC9kYzItY2xpZW50MjAKBggqhkjOPQQDAgNIADBFAiEA84vmPUZlnEmo
abauAoooNs+o+Jf0rcjHsAuaQjYS+KMCICrP91h0l3w1gI2Mw8KhVgZArm2mO2/p
KTbtt3e1FTSl
-----END CERTIFICATE-----
1 s:
i:CN = pri-7591u39p.consul.ca.43775946.consul
-----BEGIN CERTIFICATE-----
MIIB5DCCAYqgAwIBAgIBCjAKBggqhkjOPQQDAjAxMS8wLQYDVQQDEyZwcmktNzU5
MXUzOXAuY29uc3VsLmNhLjQzNzc1OTQ2LmNvbnN1bDAeFw0yMzAyMDgxNDQ5MzRa
Fw0yNDAyMDgxNDQ5MzRaMAAwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAASYy150
hIvw7ESsmEGssCq1ErR15uxuLeFQkzUiZhqZb364MYkOh0lglavOodlUSqiiNef0
pyk74S23/w6h/GZ3o4HDMIHAMA4GA1UdDwEB/wQEAwIBhjASBgNVHRMBAf8ECDAG
AQH/AgEAMCkGA1UdDgQiBCDy4oluTlcIB+o+MmKiHcsZe/OxaEZEyihgZ+vz4W6t
WDArBgNVHSMEJDAigCA8XCBqtqATMLCwDeniB5N3Q/IR+ZLyaEdiUoB1uEbqczBC
BgNVHREBAf8EODA2hjRzcGlmZmU6Ly80Mzc3NTk0Ni0wYzc2LTViZGItODcwMi1j
OTBhOGM1MjdkYjUuY29uc3VsMAoGCCqGSM49BAMCA0gAMEUCIQDN5JonU1Abl0P3
yjvsq4Qqah6SVDioxsY03Qgejtv8DAIgdRNyUwR6SHyhgkD3uFgQ6wVvMXevg9WD
/e8aHjFxV+8=
-----END CERTIFICATE-----
2 s:
i:CN = pri-7591u39p.consul.ca.43775946.consul
-----BEGIN CERTIFICATE-----
MIIB5DCCAYqgAwIBAgIBCjAKBggqhkjOPQQDAjAxMS8wLQYDVQQDEyZwcmktNzU5
MXUzOXAuY29uc3VsLmNhLjQzNzc1OTQ2LmNvbnN1bDAeFw0yMzAyMDgxNDQ5MzRa
Fw0yNDAyMDgxNDQ5MzRaMAAwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAASYy150
hIvw7ESsmEGssCq1ErR15uxuLeFQkzUiZhqZb364MYkOh0lglavOodlUSqiiNef0
pyk74S23/w6h/GZ3o4HDMIHAMA4GA1UdDwEB/wQEAwIBhjASBgNVHRMBAf8ECDAG
AQH/AgEAMCkGA1UdDgQiBCDy4oluTlcIB+o+MmKiHcsZe/OxaEZEyihgZ+vz4W6t
WDArBgNVHSMEJDAigCA8XCBqtqATMLCwDeniB5N3Q/IR+ZLyaEdiUoB1uEbqczBC
BgNVHREBAf8EODA2hjRzcGlmZmU6Ly80Mzc3NTk0Ni0wYzc2LTViZGItODcwMi1j
OTBhOGM1MjdkYjUuY29uc3VsMAoGCCqGSM49BAMCA0gAMEUCIQDN5JonU1Abl0P3
yjvsq4Qqah6SVDioxsY03Qgejtv8DAIgdRNyUwR6SHyhgkD3uFgQ6wVvMXevg9WD
/e8aHjFxV+8=
-----END CERTIFICATE-----
---
Server certificate
subject=

issuer=

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1878 bytes and written 357 bytes
Verification error: unable to get local issuer certificate
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 256 bit
Secure Renegotiation IS NOT supported
No ALPN negotiated
Early data was not sent
Verify return code: 20 (unable to get local issuer certificate)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_128_GCM_SHA256
Session-ID: F2EB76B191575FBB65EE655EF38C6D7AFAF46B954613C9963A135178EA6B44A9
Session-ID-ctx:
Resumption PSK: 4AEDB346FC6B7D265870215AA12A99416B4E7E519BF861E27D288EBF754EB59B
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 604800 (seconds)
TLS session ticket:
0000 - e6 6f c8 5b b9 c6 36 7b-72 de ab 98 14 fc 02 76 .o.[..6{r......v
0010 - f1 90 08 c5 69 4e f1 2e-0d 75 f4 6f eb d3 09 53 ....iN...u.o...S
0020 - b7 10 d6 4d 74 df fb 45-8a c6 5f 9c 2b 72 37 17 ...Mt..E.._.+r7.
0030 - 1c 77 84 9c 04 19 90 44-65 b5 02 b4 c6 20 48 45 .w.....De.... HE
0040 - 7a 30 e9 fd f2 25 72 1e-f8 2b 3b 8d 36 d7 17 99 z0...%r..+;.6...
0050 - bc 5d 40 24 be 93 f0 da-ec 7f 7b e1 d8 1a b4 67 .]@$......{....g
0060 - 79 85 ce db be 09 f1 65-53 d3 c7 d8 18 15 73 12 y......eS.....s.
0070 - 0f .

Start Time: 1675868200
Timeout : 7200 (sec)
Verify return code: 20 (unable to get local issuer certificate)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
```

### Consul config for both Client and Server

Client config

```bash
{
"datacenter": "${datacenter}",
%{~ if primary_datacenter != null }
"primary_datacenter": "${primary_datacenter}",
%{~ endif }
"retry_join": ["${service_dns_name}"],
"bind_addr": "{{ GetInterfaceIP \"eth0\" }}",
"client_addr": "0.0.0.0",
"data_dir": "/consul/data",
"ports": {
"http": 8500,
"https": 8501
},
"verify_incoming": false,
"verify_outgoing": true,
"verify_server_hostname": true,
"ca_file": "/consul/config/ca-cert.pem",
"auto_encrypt": {
"tls": true
}
}

```

Server config

```bash
{
"datacenter": "${datacenter}",
%{~ if primary_datacenter != null }
"primary_datacenter": "${primary_datacenter}",
"retry_join_wan": ["${retry_join_wan}"],
%{~ endif }
"server": true,
"bootstrap_expect": ${num_servers},
"retry_join": ["${service_dns_name}"],
"bind_addr": "{{ GetInterfaceIP \"eth0\" }}",
"client_addr": "0.0.0.0",
"data_dir": "/consul/data",
"ports": {
"http": 8500,
"https": 8501
},
"verify_incoming": false,
"verify_outgoing": true,
"verify_server_hostname": true,
"ca_file": "/consul/config/ca-cert.pem",
"cert_file": "/consul/config/server-cert.pem",
"key_file": "/consul/config/server-key.pem",
"auto_encrypt": {
"allow_tls": true
}
}
```

### Consul info for both Client and Server

Client info

```bash
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 2c56447e
version = 1.11.1
consul:
acl = disabled
known_servers = 3
server = false
runtime:
arch = amd64
cpu_count = 6
goroutines = 64
max_procs = 6
os = linux
version = go1.17.5
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 5
members = 5
query_queue = 0
query_time = 1

```

Server info

```bash
/ # consul info
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 2c56447e
version = 1.11.1
consul:
acl = disabled
bootstrap = false
known_datacenters = 2
leader = false
leader_addr = 10.42.0.16:8300
server = true
raft:
applied_index = 34
commit_index = 34
fsm_pending = 0
last_contact = 54.9445ms
last_log_index = 34
last_log_term = 2
last_snapshot_index = 0
last_snapshot_term = 0
latest_configuration = [{Suffrage:Voter ID:541ad05f-569f-dce7-ea92-0f83274714b5 Address:10.42.0.16:8300} {Suffrage:Voter ID:cc7bb1b8-dc14-66bb-16b8-fdebcd5d7d48 Address:10.42.0.15:8300} {Suffrage:Voter ID:0ad5e091-f420-ed5f-0ec2-20c60f42ec1b Address:10.42.0.14:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 2
runtime:
arch = amd64
cpu_count = 6
goroutines = 122
max_procs = 6
os = linux
version = go1.17.5
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 5
members = 5
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 5
members = 6
query_queue = 0
query_time = 1

```

### Operating system and Environment details

Servers: hashicorp/consul: 1.11.1
Clients: hashicorp/consul: 1.11.1

### Log Fragments

```
curl -vvv --cacert https://localhost:8501/v1/catalog/services
* Trying 127.0.0.1:8501...
* Connected to localhost (127.0.0.1) port 8501 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: client-leaf-cert-consul.pem
* CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: [NONE]
* start date: Jan 30 16:27:37 2023 GMT
* expire date: Feb 2 16:27:37 2023 GMT
* subjectAltName: host "localhost" matched cert's "localhost"
* SSL: couldn't get X509-issuer name!
* Closing connection 0
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS alert, close notify (256):

curl: (60) SSL: couldn't get X509-issuer name!
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
```

Start with the Consul Connect CA and auto_encrypt certificate path, using the documented two-datacenter reproduction and the /v1/connect/ca/roots endpoint. Inspect the secondary-DC chain with openssl s_client, then verify with curl; done means the client certificate has a non-empty issuer and TLS verification succeeds for the reported clients.

Written by the indexing model from the issue text.