When the vault server's root CA isn't installed, Nomad fails silently
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
0.10.3
### Operating system and Environment details
#### OS
**Distributor ID**: Ubuntu
**Description**: Ubuntu 16.04.6 LTS
**Release**: 16.04
**Codename**: xenial
### Issue
When creating a new Nomad client, if the configured vault isn't accessible with trustworthy TLS, the vault configuration will silently fail. We can see in the logs that the fingerprint runs but there is no error, even with the logs set to Debug.
Installing the root CA will fix this upon restart of the client, and we can see this by looking at Nomad server's UI and seeing:
| vault | |
| -----------------|-------------|
| vault.version | 1.1.3 |
| vault.accessible | true |
### Reproduction steps
#### Install and configure Consul
```
#!/bin/bash
mkdir -p consul/config consul/data
echo "datacenter = \"mgt1\"
encrypt = \"************\"" > consul/config/consul.hcl
IP=10.10.10.10
docker run -d --net=host --restart unless-stopped --name consul -v $(pwd)/consul/config:/consul/config -v $(pwd)/consul/data:/consul/data consul:1.5 agent -bind $IP -client $IP -join nomad.url.example.com
```
#### Install and configure Nomad
```
#!/bin/bash
apt-get install -y unzip
wget "https://releases.hashicorp.com/nomad/0.10.3/nomad_0.10.3_linux_amd64.zip"
unzip nomad_0.10.3_linux_amd64.zip
rm nomad_0.10.3_linux_amd64.zip
mkdir -p nomad_dir/config nomad_dir/data
echo "# Increase log verbosity
log_level = \"DEBUG\"
datacenter = \"mgt1\"
region = \"europe-west\"
# Setup data dir
data_dir = \"$(pwd)/nomad_dir/data\"
acl {
enabled = true
}
consul {
address = \"10.10.10.10:8500\"
}
vault {
enabled = true
address = \"https://vault.url.example.com\"
}
plugin \"docker\" {
config {
auth {
config = \"$(pwd)/nomad_dir/docker.json\"
}
}
}
# Enable the server
client {
enabled = true
}" > nomad_dir/config/client.hcl
./nomad agent -config nomad_dir/config/client.hcl 2>&1 > /var/log/nomad &
```
### Nomad Client logs
```
==> Loaded configuration from nomad_dir/config/client.hcl
==> Starting Nomad agent...
==> Nomad agent configuration:
Advertise Addrs: HTTP: 10.0.2.15:4646
Bind Addrs: HTTP: 0.0.0.0:4646
Client: true
Log Level: DEBUG
Region: europe-west (DC: mgt1)
Server: false
Version: 0.10.3
==> Nomad agent started! Log data will stream in below:
2020-02-06T16:57:28.923Z [WARN] agent.plugin_loader: skipping external plugins since plugin_dir doesn't exist: plugin_dir=/home/vagrant/nomad_dir/data/plugins
2020-02-06T16:57:28.927Z [DEBUG] agent.plugin_loader.docker: using client connection initialized from environment: plugin_dir=/home/vagrant/nomad_dir/data/plugins
2020-02-06T16:57:28.927Z [DEBUG] agent.plugin_loader.docker: using client connection initialized from environment: plugin_dir=/home/vagrant/nomad_dir/data/plugins
2020-02-06T16:57:28.927Z [INFO] agent: detected plugin: name=java type=driver plugin_version=0.1.0
2020-02-06T16:57:28.927Z [INFO] agent: detected plugin: name=docker type=driver plugin_version=0.1.0
2020-02-06T16:57:28.927Z [INFO] agent: detected plugin: name=rkt type=driver plugin_version=0.1.0
2020-02-06T16:57:28.927Z [INFO] agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
2020-02-06T16:57:28.927Z [INFO] agent: detected plugin: name=exec type=driver plugin_version=0.1.0
2020-02-06T16:57:28.927Z [INFO] agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
2020-02-06T16:57:28.927Z [INFO] agent: detected plugin: name=nvidia-gpu type=device plugin_version=0.1.0
2020-02-06T16:57:28.927Z [INFO] client: using state directory: state_dir=/home/vagrant/nomad_dir/data/client
2020-02-06T16:57:28.927Z [INFO] client: using alloc directory: alloc_dir=/home/vagrant/nomad_dir/data/alloc
2020-02-06T16:57:28.929Z [DEBUG] client.fingerprint_mgr: built-in fingerprints: fingerprinters=[arch, cgroup, consul, cpu, host, memory, network, nomad, signal, storage, vault, env_aws, env_gce]
2020-02-06T16:57:28.929Z [INFO] client.fingerprint_mgr.cgroup: cgroups are available
2020-02-06T16:57:28.929Z [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=cgroup period=15s
2020-02-06T16:57:28.931Z [INFO] client.fingerprint_mgr.consul: consul agent is available
2020-02-06T16:57:28.931Z [DEBUG] client.fingerprint_mgr.cpu: detected cpu frequency: MHz=2399
2020-02-06T16:57:28.931Z [DEBUG] client.fingerprint_mgr.cpu: detected core count: cores=2
2020-02-06T16:57:28.931Z [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=consul period=15s
2020-02-06T16:57:28.933Z [DEBUG] client.fingerprint_mgr.network: link speed detected: interface=enp0s3 mbits=1000
2020-02-06T16:57:28.933Z [DEBUG] client.fingerprint_mgr.network: detected interface IP: interface=enp0s3 IP=10.0.2.15
2020-02-06T16:57:32.787Z [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=vault period=15s
2020-02-06T16:57:34.793Z [DEBUG] client.fingerprint_mgr.env_aws: error querying AWS Metadata URL, skipping
2020-02-06T16:57:36.793Z [DEBUG] client.fingerprint_mgr.env_gce: could not read value for attribute: attribute=machine-type error="Get http://169.254.169.254/computeMetadata/v1/instance/machine-type: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
2020-02-06T16:57:36.793Z [DEBUG] client.fingerprint_mgr.env_gce: error querying GCE Metadata URL, skipping
2020-02-06T16:57:36.793Z [DEBUG] client.fingerprint_mgr: detected fingerprints: node_attrs=[arch, cgroup, consul, cpu, host, network, nomad, signal, storage]
2020-02-06T16:57:36.793Z [INFO] client.plugin: starting plugin manager: plugin-type=driver
2020-02-06T16:57:36.793Z [INFO] client.plugin: starting plugin manager: plugin-type=device
2020-02-06T16:57:36.794Z [DEBUG] client.plugin: waiting on plugin manager initial fingerprint: plugin-type=driver
2020-02-06T16:57:36.794Z [DEBUG] client.plugin: waiting on plugin manager initial fingerprint: plugin-type=device
2020-02-06T16:57:36.795Z [DEBUG] client.driver_mgr: initial driver fingerprint: driver=rkt health=undetected description="Failed to execute rkt version: exec: "rkt": executable file not found in $PATH"
2020-02-06T16:57:36.795Z [DEBUG] client.driver_mgr: initial driver fingerprint: driver=java health=undetected description=
2020-02-06T16:57:36.795Z [DEBUG] client.driver_mgr: initial driver fingerprint: driver=exec health=healthy description=Healthy
2020-02-06T16:57:36.795Z [DEBUG] client.driver_mgr: initial driver fingerprint: driver=qemu health=undetected description=
2020-02-06T16:57:36.795Z [DEBUG] client.driver_mgr: initial driver fingerprint: driver=raw_exec health=undetected description=disabled
2020-02-06T16:57:36.795Z [DEBUG] client.plugin: finished plugin manager initial fingerprint: plugin-type=device
2020-02-06T16:57:36.805Z [DEBUG] client.driver_mgr.docker: bridge_ip could not be discovered: driver=docker
2020-02-06T16:57:36.816Z [DEBUG] client.driver_mgr: initial driver fingerprint: driver=docker health=healthy description=Healthy
2020-02-06T16:57:36.816Z [DEBUG] client.driver_mgr: detected drivers: drivers="map[healthy:[exec docker] undetected:[rkt java qemu raw_exec]]"
2020-02-06T16:57:36.816Z [DEBUG] client.plugin: finished plugin manager initial fingerprint: plugin-type=driver
2020-02-06T16:57:36.816Z [INFO] client: started client: node_id=85e0c8e3-cb9e-4305-4730-4921f91c0cd5
2020-02-06T16:57:36.818Z [WARN] client.server_mgr: no servers available
2020-02-06T16:57:36.818Z [WARN] client.server_mgr: no servers available
2020-02-06T16:57:36.818Z [DEBUG] client: registration waiting on servers
2020-02-06T16:57:36.851Z [DEBUG] client.consul: bootstrap contacting Consul DCs: consul_dcs=[mgt1]
2020-02-06T16:57:37.022Z [INFO] client.consul: discovered following servers: servers=[10.97.0.5:4647]
2020-02-06T16:57:37.022Z [DEBUG] client.server_mgr: new server list: new_servers=[10.97.0.5:4647] old_servers=[]
2020-02-06T16:57:37.136Z [DEBUG] client: updated allocations: index=132331 total=0 pulled=0 filtered=0
2020-02-06T16:57:37.136Z [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=0
2020-02-06T16:57:37.137Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=0 errors=0
2020-02-06T16:57:37.149Z [INFO] client: node registration complete
2020-02-06T16:57:40.359Z [DEBUG] client: state updated: node_status=ready
2020-02-06T16:57:44.394Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=501.674µs
2020-02-06T16:57:44.764Z [DEBUG] client: state changed, updating node and re-registering
2020-02-06T16:57:44.831Z [INFO] client: node registration complete
2020-02-06T16:57:54.396Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=292.957µs
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the client.fingerprint_mgr vault fingerprint path shown in the logs and reproduce the client startup with an untrusted Vault TLS certificate. Trace how the TLS failure is handled; done means the failure is reported rather than silently omitted and the resulting Vault availability is clear.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100