hashicorp / hashicorp/nomad

When the vault server's root CA isn't installed, Nomad fails silently

Open
#7,090 4 comments 2 reactions 0 assignees View on GitHub
stage/accepted theme/vault
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
0.10.3

### Operating system and Environment details
#### OS
**Distributor ID**: Ubuntu
**Description**: Ubuntu 16.04.6 LTS
**Release**: 16.04
**Codename**: xenial

### Issue
When creating a new Nomad client, if the configured vault isn't accessible with trustworthy TLS, the vault configuration will silently fail. We can see in the logs that the fingerprint runs but there is no error, even with the logs set to Debug.
Installing the root CA will fix this upon restart of the client, and we can see this by looking at Nomad server's UI and seeing:

| vault | |
| -----------------|-------------|
| vault.version | 1.1.3 |
| vault.accessible | true |

### Reproduction steps
#### Install and configure Consul
```
#!/bin/bash

mkdir -p consul/config consul/data

echo "datacenter = \"mgt1\"
encrypt = \"************\"" > consul/config/consul.hcl

IP=10.10.10.10
docker run -d --net=host --restart unless-stopped --name consul -v $(pwd)/consul/config:/consul/config -v $(pwd)/consul/data:/consul/data consul:1.5 agent -bind $IP -client $IP -join nomad.url.example.com

```
#### Install and configure Nomad
```
#!/bin/bash

apt-get install -y unzip

wget "https://releases.hashicorp.com/nomad/0.10.3/nomad_0.10.3_linux_amd64.zip"
unzip nomad_0.10.3_linux_amd64.zip
rm nomad_0.10.3_linux_amd64.zip

mkdir -p nomad_dir/config nomad_dir/data

echo "# Increase log verbosity
log_level = \"DEBUG\"

datacenter = \"mgt1\"

region = \"europe-west\"

# Setup data dir
data_dir = \"$(pwd)/nomad_dir/data\"

acl {
enabled = true
}
consul {
address = \"10.10.10.10:8500\"
}

vault {
enabled = true
address = \"https://vault.url.example.com\"
}

plugin \"docker\" {
config {
auth {
config = \"$(pwd)/nomad_dir/docker.json\"
}
}
}

# Enable the server
client {
enabled = true
}" > nomad_dir/config/client.hcl

./nomad agent -config nomad_dir/config/client.hcl 2>&1 > /var/log/nomad &
```

### Nomad Client logs
```
==> Loaded configuration from nomad_dir/config/client.hcl
==> Starting Nomad agent...
==> Nomad agent configuration:

Advertise Addrs: HTTP: 10.0.2.15:4646
Bind Addrs: HTTP: 0.0.0.0:4646
Client: true
Log Level: DEBUG
Region: europe-west (DC: mgt1)
Server: false
Version: 0.10.3

==> Nomad agent started! Log data will stream in below:

2020-02-06T16:57:28.923Z [WARN] agent.plugin_loader: skipping external plugins since plugin_dir doesn't exist: plugin_dir=/home/vagrant/nomad_dir/data/plugins
2020-02-06T16:57:28.927Z [DEBUG] agent.plugin_loader.docker: using client connection initialized from environment: plugin_dir=/home/vagrant/nomad_dir/data/plugins
2020-02-06T16:57:28.927Z [DEBUG] agent.plugin_loader.docker: using client connection initialized from environment: plugin_dir=/home/vagrant/nomad_dir/data/plugins
2020-02-06T16:57:28.927Z [INFO] agent: detected plugin: name=java type=driver plugin_version=0.1.0
2020-02-06T16:57:28.927Z [INFO] agent: detected plugin: name=docker type=driver plugin_version=0.1.0
2020-02-06T16:57:28.927Z [INFO] agent: detected plugin: name=rkt type=driver plugin_version=0.1.0
2020-02-06T16:57:28.927Z [INFO] agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
2020-02-06T16:57:28.927Z [INFO] agent: detected plugin: name=exec type=driver plugin_version=0.1.0
2020-02-06T16:57:28.927Z [INFO] agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
2020-02-06T16:57:28.927Z [INFO] agent: detected plugin: name=nvidia-gpu type=device plugin_version=0.1.0
2020-02-06T16:57:28.927Z [INFO] client: using state directory: state_dir=/home/vagrant/nomad_dir/data/client
2020-02-06T16:57:28.927Z [INFO] client: using alloc directory: alloc_dir=/home/vagrant/nomad_dir/data/alloc
2020-02-06T16:57:28.929Z [DEBUG] client.fingerprint_mgr: built-in fingerprints: fingerprinters=[arch, cgroup, consul, cpu, host, memory, network, nomad, signal, storage, vault, env_aws, env_gce]
2020-02-06T16:57:28.929Z [INFO] client.fingerprint_mgr.cgroup: cgroups are available
2020-02-06T16:57:28.929Z [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=cgroup period=15s
2020-02-06T16:57:28.931Z [INFO] client.fingerprint_mgr.consul: consul agent is available
2020-02-06T16:57:28.931Z [DEBUG] client.fingerprint_mgr.cpu: detected cpu frequency: MHz=2399
2020-02-06T16:57:28.931Z [DEBUG] client.fingerprint_mgr.cpu: detected core count: cores=2
2020-02-06T16:57:28.931Z [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=consul period=15s
2020-02-06T16:57:28.933Z [DEBUG] client.fingerprint_mgr.network: link speed detected: interface=enp0s3 mbits=1000
2020-02-06T16:57:28.933Z [DEBUG] client.fingerprint_mgr.network: detected interface IP: interface=enp0s3 IP=10.0.2.15
2020-02-06T16:57:32.787Z [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=vault period=15s
2020-02-06T16:57:34.793Z [DEBUG] client.fingerprint_mgr.env_aws: error querying AWS Metadata URL, skipping
2020-02-06T16:57:36.793Z [DEBUG] client.fingerprint_mgr.env_gce: could not read value for attribute: attribute=machine-type error="Get http://169.254.169.254/computeMetadata/v1/instance/machine-type: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
2020-02-06T16:57:36.793Z [DEBUG] client.fingerprint_mgr.env_gce: error querying GCE Metadata URL, skipping
2020-02-06T16:57:36.793Z [DEBUG] client.fingerprint_mgr: detected fingerprints: node_attrs=[arch, cgroup, consul, cpu, host, network, nomad, signal, storage]
2020-02-06T16:57:36.793Z [INFO] client.plugin: starting plugin manager: plugin-type=driver
2020-02-06T16:57:36.793Z [INFO] client.plugin: starting plugin manager: plugin-type=device
2020-02-06T16:57:36.794Z [DEBUG] client.plugin: waiting on plugin manager initial fingerprint: plugin-type=driver
2020-02-06T16:57:36.794Z [DEBUG] client.plugin: waiting on plugin manager initial fingerprint: plugin-type=device
2020-02-06T16:57:36.795Z [DEBUG] client.driver_mgr: initial driver fingerprint: driver=rkt health=undetected description="Failed to execute rkt version: exec: "rkt": executable file not found in $PATH"
2020-02-06T16:57:36.795Z [DEBUG] client.driver_mgr: initial driver fingerprint: driver=java health=undetected description=
2020-02-06T16:57:36.795Z [DEBUG] client.driver_mgr: initial driver fingerprint: driver=exec health=healthy description=Healthy
2020-02-06T16:57:36.795Z [DEBUG] client.driver_mgr: initial driver fingerprint: driver=qemu health=undetected description=
2020-02-06T16:57:36.795Z [DEBUG] client.driver_mgr: initial driver fingerprint: driver=raw_exec health=undetected description=disabled
2020-02-06T16:57:36.795Z [DEBUG] client.plugin: finished plugin manager initial fingerprint: plugin-type=device
2020-02-06T16:57:36.805Z [DEBUG] client.driver_mgr.docker: bridge_ip could not be discovered: driver=docker
2020-02-06T16:57:36.816Z [DEBUG] client.driver_mgr: initial driver fingerprint: driver=docker health=healthy description=Healthy
2020-02-06T16:57:36.816Z [DEBUG] client.driver_mgr: detected drivers: drivers="map[healthy:[exec docker] undetected:[rkt java qemu raw_exec]]"
2020-02-06T16:57:36.816Z [DEBUG] client.plugin: finished plugin manager initial fingerprint: plugin-type=driver
2020-02-06T16:57:36.816Z [INFO] client: started client: node_id=85e0c8e3-cb9e-4305-4730-4921f91c0cd5
2020-02-06T16:57:36.818Z [WARN] client.server_mgr: no servers available
2020-02-06T16:57:36.818Z [WARN] client.server_mgr: no servers available
2020-02-06T16:57:36.818Z [DEBUG] client: registration waiting on servers
2020-02-06T16:57:36.851Z [DEBUG] client.consul: bootstrap contacting Consul DCs: consul_dcs=[mgt1]
2020-02-06T16:57:37.022Z [INFO] client.consul: discovered following servers: servers=[10.97.0.5:4647]
2020-02-06T16:57:37.022Z [DEBUG] client.server_mgr: new server list: new_servers=[10.97.0.5:4647] old_servers=[]
2020-02-06T16:57:37.136Z [DEBUG] client: updated allocations: index=132331 total=0 pulled=0 filtered=0
2020-02-06T16:57:37.136Z [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=0
2020-02-06T16:57:37.137Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=0 errors=0
2020-02-06T16:57:37.149Z [INFO] client: node registration complete
2020-02-06T16:57:40.359Z [DEBUG] client: state updated: node_status=ready
2020-02-06T16:57:44.394Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=501.674µs
2020-02-06T16:57:44.764Z [DEBUG] client: state changed, updating node and re-registering
2020-02-06T16:57:44.831Z [INFO] client: node registration complete
2020-02-06T16:57:54.396Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=292.957µs
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the client.fingerprint_mgr vault fingerprint path shown in the logs and reproduce the client startup with an untrusted Vault TLS certificate. Trace how the TLS failure is handled; done means the failure is reported rather than silently omitted and the resulting Vault availability is clear.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.