Consul rpc port communicates over tls when both http and https enabled leading to nomad mesh integration warnings
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
A cluster with consul(**v 1.9.2**) running has both http and https enabled on 8500 and 8501 respectively (for backward compatibility). RPC tls is disabled by setting verify_incoming/verify_outgoing configurations to false.
Consul server config and client config attached.
[consul_client_config.txt](https://github.com/hashicorp/consul/files/6005941/consul_client_config.txt)
[consul_server_config.txt](https://github.com/hashicorp/consul/files/6005849/consul_server_config.txt)
Nomad(**v 1.0.2**) mesh integration is enabled and has an ingress job running. Nomad is still communicating with consul over http and 8500 (consul stanza config below)
```
consul {
address = "127.0.0.1:8500"
token = "XXXX"
ssl = false
}
```
I see following warnings being logged every few seconds in consul and nomad
**Nomad logs:**
`2021-02-18T21:15:07.414Z [WARN] client.alloc_runner.runner_hook: error proxying from Consul: alloc_id=170c1860-efd9-024a-0a4d-5b2bb4d02d53 error="read tcp 127.0.0.1:54738->127.0.0.1:8502: read: connection reset by peer" dest=127.0.0.1:8502 src_local=/opt/nomad/alloc/170c1860-efd9-024a-0a4d-5b2bb4d02d53/alloc/tmp/consul_grpc.sock src_remote=@ bytes=0`
**Consul logs:**
`2021-02-18T21:15:07.393Z [WARN] agent: grpc: Server.Serve failed to complete security handshake from "127.0.0.1:54738": tls: first record does not look like a TLS handshake`
It is also observed that 8502 is now communicating over tls, checked by running command
`openssl s_client -connect NODE_IP:8502`
This command returned a certificate when consul has both http and https enabled but no certificate when only http is running.
1) Both http and https enabled, RPC configs (verify_incoming & verify_outgoing) are set to false
```
depth=0 CN = server.XXXX.consul
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = server.XXXX.consul
verify error:num=21:unable to verify the first certificate
verify return:1
Certificate chain
0 s:/CN=server.XXXX.consul
i:/C=US/ST=CA/L=San Francisco/street=101 Second Street/postalCode=94105/O=HashiCorp Inc./CN=Consul Agent CA 76243903559371484871955553276941343535
Server certificate
BEGIN CERTIFICATE
MIIC2TCCAn+gAwIBAgIRALqE9myBIai3kEoWwhxFZ8EwCgYIKoZIzj0EAwIwgbgx
CzAJBgNVBAYTAlVTMQswCQYDVQQIEwJDQTEWMBQGA1UEBxMNU2FuIEZyYW5jaXNj
bzEaMBgGA1UECRMRMTAxIFNlY29uZCBTdHJlZXQxDjAMBgNVBBETBTk0MTA1MRcw
FQYDVQQKEw5IYXNoaUNvcnAgSW5jLjE/MD0GA1UEAxM2Q29uc3VsIEFnZW50IENB
IDc2MjQzOTAzNTU5MzcxNDg0ODcxOTU1NTUzMjc2OTQxMzQzNTM1MB4XDTIxMDIw
MjEzNTUyMlGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwKQYDVR0OBCIEIOnJ
hYCvHS3ujBiE1HrkiXsEt7DCIcg9M4kz5jVyoVSpMCsGA1UdIwQkMCKAIAZBod1y
rec3Fl9IzTY/CBMMyyA6006svfyDHD0SDsLEMFoGA1UdEQRTMFGCFWNvbnN1bC5z
ZXJ2aWNlLmNvbnXXXX4bWXUG2tACmQExHAiBeI2cwKFJcjm7RhQMqeblz7itB
jSNCoUB8TvDYFIteRA==
END CERTIFICATE
subject=/CN=server.XXXX.consul
issuer=/C=US/ST=CA/L=San Francisco/street=101 Second Street/postalCode=94105/O=HashiCorp Inc./CN=Consul Agent CA 76243903559371484871955553276941343535
No client certificate CA names sent
Server Temp Key: ECDH, X25519, 253 bits
SSL handshake has read 1136 bytes and written 289 bytes
```
2) Only http enabled:
```
l-47.140.1/libressl-2.8/ssl/ssl_pkt.c:386:
no peer certificate available
No client certificate CA names sent
SSL handshake has read 5 bytes and written 0 bytes
```
Steps to reproduce this issue:
1. Create a 3 node consul cluster with connect enabled and both http & https running.
2. Enable nomad, consul integration with consul http url.
3. Deploy a job and the warnings should be seen in consul and nomad logs.
### Consul info for both Client and Server
```
agent:
check_monitors = 1
check_ttls = 5
checks = 16
services = 17
build:
prerelease =
revision = 6530cf37
version = 1.9.2
consul:
acl = enabled
bootstrap = false
known_datacenters = 1
leader = false
leader_addr = 10.84.36.27:8300
server = true
raft:
applied_index = 8124210
commit_index = 8124210
fsm_pending = 0
last_contact = 53.075348ms
last_log_index = 8124210
last_log_term = 75
last_snapshot_index = 8115331
last_snapshot_term = 74
latest_configuration = [{Suffrage:Voter ID:c3f74a34-feea-53ea-b739-58d639c20333 Address:10.84.37.225:8300} {Suffrage:Voter ID:bc0cd832-6dae-8217-e41f-e834f5956636 Address:10.84.36.26:8300} {Suffrage:Voter ID:ed0c8cdc-d758-0e7b-bde0-3420d05e9b84 Address:10.84.36.27:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 75
runtime:
arch = amd64
cpu_count = 8
goroutines = 177
max_procs = 8
os = linux
version = go1.15.6
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 37
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 6424
members = 4
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 60
members = 3
query_queue = 0
query_time = 1
```
Contributor guide
Research direction
Begin by reviewing the attached Consul client and server configurations, the RPC TLS settings, and the Nomad consul stanza, then reproduce the three-node setup with both HTTP and HTTPS enabled. Compare the Consul and Nomad logs with the openssl checks on port 8502; done means RPC traffic respects the configured TLS settings without repeated handshake or connection-reset warnings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100