hashicorp / hashicorp/consul

Consul rpc port communicates over tls when both http and https enabled leading to nomad mesh integration warnings

Open
#9,791 1 comment 3 reactions 0 assignees View on GitHub
theme/connect theme/consul-nomad theme/internals type/bug
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

A cluster with consul(**v 1.9.2**) running has both http and https enabled on 8500 and 8501 respectively (for backward compatibility). RPC tls is disabled by setting verify_incoming/verify_outgoing configurations to false.
Consul server config and client config attached.
[consul_client_config.txt](https://github.com/hashicorp/consul/files/6005941/consul_client_config.txt)
[consul_server_config.txt](https://github.com/hashicorp/consul/files/6005849/consul_server_config.txt)

Nomad(**v 1.0.2**) mesh integration is enabled and has an ingress job running. Nomad is still communicating with consul over http and 8500 (consul stanza config below)
```
consul {
address = "127.0.0.1:8500"
token = "XXXX"
ssl = false
}
```

I see following warnings being logged every few seconds in consul and nomad
**Nomad logs:**
`2021-02-18T21:15:07.414Z [WARN] client.alloc_runner.runner_hook: error proxying from Consul: alloc_id=170c1860-efd9-024a-0a4d-5b2bb4d02d53 error="read tcp 127.0.0.1:54738->127.0.0.1:8502: read: connection reset by peer" dest=127.0.0.1:8502 src_local=/opt/nomad/alloc/170c1860-efd9-024a-0a4d-5b2bb4d02d53/alloc/tmp/consul_grpc.sock src_remote=@ bytes=0`

**Consul logs:**
`2021-02-18T21:15:07.393Z [WARN] agent: grpc: Server.Serve failed to complete security handshake from "127.0.0.1:54738": tls: first record does not look like a TLS handshake`

It is also observed that 8502 is now communicating over tls, checked by running command
`openssl s_client -connect NODE_IP:8502`
This command returned a certificate when consul has both http and https enabled but no certificate when only http is running.

1) Both http and https enabled, RPC configs (verify_incoming & verify_outgoing) are set to false
```
depth=0 CN = server.XXXX.consul
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = server.XXXX.consul
verify error:num=21:unable to verify the first certificate
verify return:1

Certificate chain
0 s:/CN=server.XXXX.consul
i:/C=US/ST=CA/L=San Francisco/street=101 Second Street/postalCode=94105/O=HashiCorp Inc./CN=Consul Agent CA 76243903559371484871955553276941343535

Server certificate
BEGIN CERTIFICATE
MIIC2TCCAn+gAwIBAgIRALqE9myBIai3kEoWwhxFZ8EwCgYIKoZIzj0EAwIwgbgx
CzAJBgNVBAYTAlVTMQswCQYDVQQIEwJDQTEWMBQGA1UEBxMNU2FuIEZyYW5jaXNj
bzEaMBgGA1UECRMRMTAxIFNlY29uZCBTdHJlZXQxDjAMBgNVBBETBTk0MTA1MRcw
FQYDVQQKEw5IYXNoaUNvcnAgSW5jLjE/MD0GA1UEAxM2Q29uc3VsIEFnZW50IENB
IDc2MjQzOTAzNTU5MzcxNDg0ODcxOTU1NTUzMjc2OTQxMzQzNTM1MB4XDTIxMDIw
MjEzNTUyMlGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwKQYDVR0OBCIEIOnJ
hYCvHS3ujBiE1HrkiXsEt7DCIcg9M4kz5jVyoVSpMCsGA1UdIwQkMCKAIAZBod1y
rec3Fl9IzTY/CBMMyyA6006svfyDHD0SDsLEMFoGA1UdEQRTMFGCFWNvbnN1bC5z
ZXJ2aWNlLmNvbnXXXX4bWXUG2tACmQExHAiBeI2cwKFJcjm7RhQMqeblz7itB
jSNCoUB8TvDYFIteRA==
END CERTIFICATE
subject=/CN=server.XXXX.consul
issuer=/C=US/ST=CA/L=San Francisco/street=101 Second Street/postalCode=94105/O=HashiCorp Inc./CN=Consul Agent CA 76243903559371484871955553276941343535

No client certificate CA names sent
Server Temp Key: ECDH, X25519, 253 bits
SSL handshake has read 1136 bytes and written 289 bytes
```

2) Only http enabled:
```
l-47.140.1/libressl-2.8/ssl/ssl_pkt.c:386:

no peer certificate available

No client certificate CA names sent

SSL handshake has read 5 bytes and written 0 bytes
```

Steps to reproduce this issue:

1. Create a 3 node consul cluster with connect enabled and both http & https running.
2. Enable nomad, consul integration with consul http url.
3. Deploy a job and the warnings should be seen in consul and nomad logs.

### Consul info for both Client and Server
```
agent:
check_monitors = 1
check_ttls = 5
checks = 16
services = 17
build:
prerelease =
revision = 6530cf37
version = 1.9.2
consul:
acl = enabled
bootstrap = false
known_datacenters = 1
leader = false
leader_addr = 10.84.36.27:8300
server = true
raft:
applied_index = 8124210
commit_index = 8124210
fsm_pending = 0
last_contact = 53.075348ms
last_log_index = 8124210
last_log_term = 75
last_snapshot_index = 8115331
last_snapshot_term = 74
latest_configuration = [{Suffrage:Voter ID:c3f74a34-feea-53ea-b739-58d639c20333 Address:10.84.37.225:8300} {Suffrage:Voter ID:bc0cd832-6dae-8217-e41f-e834f5956636 Address:10.84.36.26:8300} {Suffrage:Voter ID:ed0c8cdc-d758-0e7b-bde0-3420d05e9b84 Address:10.84.36.27:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 75
runtime:
arch = amd64
cpu_count = 8
goroutines = 177
max_procs = 8
os = linux
version = go1.15.6
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 37
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 6424
members = 4
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 60
members = 3
query_queue = 0
query_time = 1
```

Contributor guide

Open the contributing guide

Research direction

Begin by reviewing the attached Consul client and server configurations, the RPC TLS settings, and the Nomad consul stanza, then reproduce the three-node setup with both HTTP and HTTPS enabled. Compare the Consul and Nomad logs with the openssl checks on port 8502; done means RPC traffic respects the configured TLS settings without repeated handshake or connection-reset warnings.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.