hashicorp / hashicorp/nomad

auto_config does not work out of box with tls and service mesh

Open
#14,458 1 comment 0 reactions 0 assignees View on GitHub
theme/docs
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
Nomad v1.3.5

### Cunsul version
Consul v1.13.1

I try out the nomad ( consul ) service mesh capabilities. I try out the [counter-dashboard](https://www.nomadproject.io/docs/integrations/consul-connect) example.
Without TLS enabled it works as expected. So it counts :-)

After the first success of deployment I follow the [guide for consul security](https://learn.hashicorp.com/tutorials/consul/gossip-encryption-secure?in=consul/security).

After this step the envoy proxy is not able to connect the service. The envoy proxy says:

> [2022-09-02 19:54:31.353][1][warning][config] [./source/common/config/grpc_stream.h:196] DeltaAggregatedResources gRPC config stream closed since 34s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination

![image](https://user-images.githubusercontent.com/5046525/188227504-3b464193-85ee-48ff-ab8e-e87a9264f23d.png)

I use cfssl for generating my certificates and not use the consul provided one. but I try the dasboard example with the ca tool of consul as well. Both results are the same.

After some digging I find a workarround.

The [auto_config section ](https://learn.hashicorp.com/tutorials/consul/gossip-encryption-secure?in=consul/security) define the client cert section as shown below.
```hcl
"verify_incoming": false,
"verify_outgoing": true,
"verify_server_hostname": true,
"ca_file": "/consul/config/certs/consul-agent-ca.pem"
```

But as you can see below, I put there my own generated certificates.
and add env vars to the nomad.service systemd file
```systemd
Environment="CONSUL_HTTP_ADDR=127.0.0.1:8501"
Environment="CONSUL_HTTP_SSL=true"
```

Without adding the confusing and unessarry cert_file and key_file definition in consul client config I got an error from nomad client

> 2022-09-02T20:23:02.422Z [ERROR] client: error discovering nomad servers: error=\"client.consul: unable to query Consul datacenters: Get \\\"https://127.0.0.1:8501/v1/catalog/datacenters\\\": x509: certificate signed by unknown authority\""

Without adding systemd env vars the proxy is not connecting.

Is something lacking in the docu or did I miss something ?

For a simple acl I use the management token 'e95b599e-166e-7d80-08ad-aee76e7ddf19'

Consul server:
```json
{
"node_name": "master-01",
"datacenter": "nomadder_1",
"data_dir": "/opt/deployments/core/consul/data",
"server": true,
"log_level" : "INFO",
"bind_addr": "0.0.0.0",
"advertise_addr": "10.21.21.41",
"client_addr": "0.0.0.0",
"encrypt": "G1CHAD7wwu0tU28BlKkirSahTJ/Tqpo9ClOAycQAUwE=",
"ui_config": {
"enabled" : true
},
"addresses": {
"grpc" : "127.0.0.1"
},
"ports": {
"grpc" : 8502,
"https": 8501
},
"connect": {
"enabled": true
},
"retry_join":["10.21.21.41"],
"bootstrap_expect": 1,
"acl": {
"enabled": true,
"default_policy": "deny",
"enable_token_persistence": true,
"tokens": {
"initial_management": "e95b599e-166e-7d80-08ad-aee76e7ddf19",
"agent": "e95b599e-166e-7d80-08ad-aee76e7ddf19"
}
},
"performance" :{
"raft_multiplier" : 1
},
"auto_config": {
"authorization": {
"enabled": true,
"static": {
"jwt_validation_pub_keys": ["-----BEGIN PUBLIC KEY-----\nMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEjv+sWRoGN8SDkNuPxRoKpJ3YRj90\n+2g9VW9eBwNkxRM0womm13NqY/tTA9VYWTWFOV5JAhAA6mFGinNGdOkcLA==\n-----END PUBLIC KEY-----\n"],
"bound_issuer": "secint",
"bound_audiences": ["consul-cluster-nomadder_1"],
"claim_mappings": {
"sub": "node_name"
},
"claim_assertions": [
"value.node_name == \"${node}\""
]
}
}
},
"node_meta": {
"node_type": "server"
},
"tls":{
"defaults":{
"ca_file": "/usr/local/share/ca-certificates/cloudlocal/cluster-ca-bundle.pem",
"cert_file": "/etc/opt/certs/consul/consul.pem",
"key_file": "/etc/opt/certs/consul/consul-key.pem",
"verify_incoming": true,
"verify_outgoing": true
},
"internal_rpc":{
"verify_server_hostname": true
}
}
}
```

nomad server

```hcl
log_level = "DEBUG"
name = "master-01"
datacenter = "nomadder_1"
data_dir = "/opt/deployments/core/nomad/data"

server {
enabled = true
bootstrap_expect = 1
encrypt = "4PRfoE6Mj9dHTLpnzmYD1+THdlyAo2Ji4U6ewMumpAw="
}

tls {
http = true
rpc = true

ca_file = "/usr/local/share/ca-certificates/cloudlocal/cluster-ca-bundle.pem"
cert_file = "/etc/opt/certs/nomad/nomad.pem"
key_file = "/etc/opt/certs/nomad/nomad-key.pem"

verify_server_hostname = true
verify_https_client = false
}

consul{
token = "e95b599e-166e-7d80-08ad-aee76e7ddf19"
allow_unauthenticated = false
}

ui {
enabled = true

consul {
ui_url = "http://10.21.21.41:8500/ui"
}

vault {
ui_url = "https://10.21.21.41:8200/ui"
}
}
```

Consul client

```json
{
"node_name": "worker-01",
"datacenter": "nomadder_1",
"data_dir": "/opt/deployments/core/consul/data",
"ports": {"https":8501},
"bind_addr": "0.0.0.0",
"advertise_addr": "10.21.21.42",
"auto_config":{
"enabled": true,
"intro_token_file": "/opt/deployments/core/consul/data/tokens/jwt/worker-01.jwt",
"server_addresses":["10.21.21.41"]
},
"node_meta": {
"node_type": "worker"
},
"ports": {
"grpc" : 8502,
"https": 8501
},
"connect": {
"enabled": true
},
"tls":{
"defaults":{
"ca_file": "/usr/local/share/ca-certificates/cloudlocal/cluster-ca-bundle.pem",
"cert_file": "/etc/opt/certs/consul/consul.pem",
"key_file": "/etc/opt/certs/consul/consul-key.pem",
"verify_incoming": false,
"verify_outgoing": true
},
"internal_rpc":{
"verify_server_hostname": true
}
}
}
```

Nomad client

```hcl
log_level = "DEBUG"
name = "worker-01"
datacenter = "nomadder_1"
data_dir = "/opt/deployments/core/nomad/data"

client {
enabled = true
}

tls {
http = true
rpc = true

ca_file = "/usr/local/share/ca-certificates/cloudlocal/cluster-ca-bundle.pem"
cert_file = "/etc/opt/certs/nomad/nomad.pem"
key_file = "/etc/opt/certs/nomad/nomad-key.pem"

verify_server_hostname = true
verify_https_client = true
}

consul{
token = "e95b599e-166e-7d80-08ad-aee76e7ddf19"
allow_unauthenticated = false
}

plugin "docker" {
config {
volumes {
enabled = true
}
}
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the counter-dashboard example and the documented auto_config and Consul security guides, then compare the Nomad and Consul TLS settings with the systemd CONSUL_HTTP_ADDR and CONSUL_HTTP_SSL environment variables. Reproduce the Envoy connection failure with TLS enabled; done means the example works with the documented configuration or the required certificate settings are clarified.

Written by the indexing model from the issue text.

Assessment

Domain
backend-api-design, networking, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.