hashicorp / hashicorp/consul

Configure Envoy TLS accepted certificate signatures algorithms

Open
#17,176 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue
It's not trivial to configure Envoy accepted signatures.

I'm setting up a Nomad cluster, when using Consul Connect with a
`sidecar_service` Envoy could not communicate with Consul.

This was initially reported as a bug because I'm dumb, I'm leaving it here so people can see how to configure Envoy accepted certificate signatures in the workaround section (though you should use RSA instead).

---

#### Reproduction Steps
1. Using the hashicorp Debian repository, setup Nomad+Consul using ed25519-signed certs.
2. Attempt to use `connect { sidecar_service {} }`.
3. Envoy fails to reach Consul gRPC with `SSLV3_ALERT_HANDSHAKE_FAILURE`

#### Workaround
1. Use envoy v1.26.0 so you can set `signature_algorithms` in `tls_params`,
when using Nomad this is done by setting the `connect.sidecar_image` and
`connect.gateway_image` metas.
2. Copy the entire `boostrap_envoy.json` template from [bootstrap_tpl.go](https://github.com/hashicorp/consul/blob/v1.15.2/command/connect/envoy/bootstrap_tpl.go#L158-L304)
3. In the template, set the client algorithms in `common_tls_context` with a
list that overlaps what Consul uses:
```diff
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.>
"common_tls_context": {
+ "tls_params": {
+ "signature_algorithms": [
+ "ed25519"
+ ]
+ },
"validation_context": {
"trusted_ca": {
"inline_string": "{{ .AgentCAPEM }}"
```

4. In every connect block, override the config.
```
connect {
sidecar_service {
proxy {
config {
envoy_bootstrap_json_tpl = var.envoy_bootstrap_tpl
}
}
}
}
```

Envoy TLS configuration should probably be exposed to Consul (which would allow me to enable mTLS but that's another topic).

### Consul info for both Client and Server

Client info

```
agent:
check_monitors = 0
check_ttls = 1
checks = 10
services = 8
build:
prerelease =
revision = 5e08e229
version = 1.15.2
version_metadata =
consul:
acl = disabled
known_servers = 3
server = false
runtime:
arch = amd64
cpu_count = 2
goroutines = 131
max_procs = 2
os = linux
version = go1.20.1
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 33
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 53605
members = 9
query_queue = 0
query_time = 1
```

```
connect {
enabled = true
}

tls {
defaults {
verify_incoming = true
verify_outgoing = true

ca_file = "…"
cert_file = "…"
key_file = "…"
}

grpc {
verify_incoming = false
}

internal_rpc {
verify_server_hostname = true
}
}
```

Server info

```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 5e08e229
version = 1.15.2
version_metadata =
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = true
leader_addr = x.x.x.x:8300
server = true
raft:
applied_index = 603699
commit_index = 603699
fsm_pending = 0
last_contact = 0
last_log_index = 603699
last_log_term = 71
last_snapshot_index = 589960
last_snapshot_term = 71
latest_configuration = [{Suffrage:Voter ID:ca4accb7-2d7c-7a1a-7683-77ecbdc40ba9 Address:x.x.x.x:8300} {Suffrage:Voter ID:448dde0b-f98a-cf8a-d5b0-82d9aed69dd3 Address:x.x.x.x:8300} {Suffrage:Voter ID:1c78c749-75f9-e266-1ebd-bb7f0da701d1 Address:x.x.x.x:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 71
runtime:
arch = amd64
cpu_count = 2
goroutines = 297
max_procs = 2
os = linux
version = go1.20.1
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 33
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 53605
members = 9
query_queue = 0
query_time = 1
```

```
auto_encrypt {
allow_tls = true
}

tls {
defaults {
verify_incoming = true
verify_outgoing = true

ca_file = "…"
cert_file = "…"
key_file = "…"
}

internal_rpc {
verify_server_hostname = true
}

https {
ca_file = "…"
cert_file = "…"
key_file = "…"

verify_incoming = false
}
}
```

### Operating system and Environment details
- Kernel: `Linux 5.10.0-21-cloud-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64 GNU/Linux`
- OS: `Debian GNU/Linux 11 (bullseye)`
- Packages from: `deb [arch=amd64] https://apt.releases.hashicorp.com bullseye main`
- ```
$ consul -version
Consul v1.15.2
Revision 5e08e229
Build Date 2023-03-30T17:51:19Z
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)
```
- ```
$ nomad -version
Nomad v1.5.3
BuildDate 2023-04-04T20:09:50Z
Revision 434f7a1745c6304d607562daa9a4a635def7153f
```
- The Envoy docker image tag used in `${NOMAD_envoy_version}` is `v1.25.1`, I'm
guessing this is decided by Consul but could not find where to change it.

### Log Fragments
#### Envoy (log_level debug)
```
[2023-04-28 10:49:49.835][1][debug][pool] [source/common/http/conn_pool_base.cc:78] queueing stream due to no available connections (ready=0 busy=0 connecting=0)
[2023-04-28 10:49:49.835][1][debug][pool] [source/common/conn_pool/conn_pool_base.cc:291] trying to create new connection
[2023-04-28 10:49:49.835][1][debug][pool] [source/common/conn_pool/conn_pool_base.cc:145] creating a new connection (connecting=0)
[2023-04-28 10:49:49.835][1][debug][http2] [source/common/http/http2/codec_impl.cc:1586] [C5] updating connection-level initial window size to 268435456
[2023-04-28 10:49:49.835][1][debug][connection] [./source/common/network/connection_impl.h:98] [C5] current connecting state: true
[2023-04-28 10:49:49.835][1][debug][client] [source/common/http/codec_client.cc:57] [C5] connecting
[2023-04-28 10:49:49.835][1][debug][connection] [source/common/network/connection_impl.cc:941] [C5] connecting to alloc/tmp/consul_grpc.sock
[2023-04-28 10:49:49.836][1][debug][connection] [source/common/network/connection_impl.cc:688] [C5] connected
[2023-04-28 10:49:49.837][1][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:360] [C5] Async cert validation completed
[2023-04-28 10:49:49.837][1][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:234] [C5] remote address:alloc/tmp/consul_grpc.sock,TLS error: 268436496:SSL routines:OPENSSL_internal:SSLV3_ALERT_HANDSHAKE_FAILURE
[2023-04-28 10:49:49.837][1][debug][connection] [source/common/network/connection_impl.cc:250] [C5] closing socket: 0
[2023-04-28 10:49:49.837][1][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:234] [C5] remote address:alloc/tmp/consul_grpc.sock,TLS error: 268436496:SSL routines:OPENSSL_internal:SSLV3_ALERT_HANDSHAKE_FAILURE
[2023-04-28 10:49:49.837][1][debug][client] [source/common/http/codec_client.cc:107] [C5] disconnect. resetting 0 pending requests
[2023-04-28 10:49:49.837][1][debug][pool] [source/common/conn_pool/conn_pool_base.cc:484] [C5] client disconnected, failure reason: TLS error: 268436496:SSL routines:OPENSSL_internal:SSLV3_ALERT_HANDSHAKE_FAILURE
[2023-04-28 10:49:49.837][1][debug][router] [source/common/router/router.cc:1278] [C0][S8602616725729965611] upstream reset: reset reason: connection failure, transport failure reason: TLS error: 268436496:SSL routines:OPENSSL_internal:SSLV3_ALERT_HANDSHAKE_FAILURE
[2023-04-28 10:49:49.837][1][debug][http] [source/common/http/async_client_impl.cc:123] async http request response headers (end_stream=true):
':status', '200'
'content-type', 'application/grpc'
'grpc-status', '14'
'grpc-message', 'upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436496:SSL routines:OPENSSL_internal:SSLV3_ALERT_HANDSHAKE_FAILURE'

[2023-04-28 10:49:49.837][1][debug][config] [./source/common/config/grpc_stream.h:197] DeltaAggregatedResources gRPC config stream to local_agent closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436496:SSL routines:OPENSSL_internal:SSLV3_ALERT_HANDSHAKE_FAILURE
[2023-04-28 10:49:49.837][1][debug][config] [source/common/config/grpc_subscription_impl.cc:115] gRPC update for type.googleapis.com/envoy.config.cluster.v3.Cluster failed
```

#### `consul monitor --log-level trace`
```
2023-04-28T10:51:38.490Z [TRACE] agent: [core][Server #2] grpc: Server.Serve failed to create ServerTransport: connection error: desc = "ServerHandshake(\"127.0.0.1:51890\") failed: tls: peer doesn't support any of the certificate's signature algorithms"
```

#### Side to side _Client Hello_ and _Server Hello_:

![tls](https://user-images.githubusercontent.com/947312/235135031-67829b7a-3f40-4141-b6ec-6169ee66c72e.png)

Contributor guide

Open the contributing guide

Research direction

Read command/connect/envoy/bootstrap_tpl.go and trace how sidecar_service proxy configuration selects the Envoy bootstrap template. Compare the current template with the documented tls_params workaround and establish the intended configuration path for accepted certificate signature algorithms. Done should include a maintainer-approved Consul configuration approach and documentation of its limits.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
networking, security
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.