hashicorp / hashicorp/nomad

Templates secrets are read from Vault when Vault integration isn't enabled

Open
#5,262 0 comments 0 reactions 0 assignees View on GitHub
theme/consul-template theme/vault type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
```
Nomad v0.8.7 (21a2d93eecf018ad2209a5eab6aae6c359267933)
```

Tested against latest master too

### Operating system and Environment details
macOS

### Issue
If a job task has a template stanza referencing Vault, Nomad would make a Vault call even if Nomad Vault integration is not configured. In such case, Vault would make unauthenticated API calls. These calls will fail (either through connection error, or permission error), and they will get retried multiple times.

### Reproduction steps

1. Start a nomad dev agent `nomad agent --dev` (or a non-dev agent too)
2. Submit job spec below `nomad job run ./example-vault.hcl`
3. Monitor nomad and alloc

I expect task to fail start immediately, but see many references to reading with exponential backoff. You can spin up `nc -l 8200` server to dump the request API.

### Nomad logs (if appropriate)

```
==> No configuration files loaded
==> Starting Nomad agent...
==> Nomad agent configuration:

Advertise Addrs: HTTP: 127.0.0.1:4646; RPC: 127.0.0.1:4647; Serf: 127.0.0.1:4648
Bind Addrs: HTTP: 127.0.0.1:4646; RPC: 127.0.0.1:4647; Serf: 127.0.0.1:4648
Client: true
Log Level: DEBUG
Region: global (DC: dc1)
Server: true
Version: 0.8.7

==> Nomad agent started! Log data will stream in below:
[...]

2019/01/29 09:18:49.032887 [DEBUG] client: starting task runners for alloc 'ed627e93-debe-834d-627b-28a11f3adea0'
2019/01/29 09:18:49.033004 [DEBUG] client.alloc_watcher: deadline for alloc "ed627e93-debe-834d-627b-28a11f3adea0" is at 2019-01-29 09:23:49.032972104 -0500 EST m=+306.896702646 (deploy=false checks=true)
2019/01/29 09:18:49.033060 [DEBUG] client: starting task context for 'redis' (alloc 'ed627e93-debe-834d-627b-28a11f3adea0')
2019/01/29 09:18:49.044822 [INFO] (runner) creating new runner (dry: false, once: false)
2019/01/29 09:18:49.045119 [DEBUG] (runner) final config: {"Consul":{"Address":"","Auth":{"Enabled":false,"Username":"","Password":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"","Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":13,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000},"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"WARN","MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0"},"Templates":[{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"supersecret = \"{{with secret \"secret/data/secret\"}}{{.Data.data.value}}{{end}}\"\n","CreateDestDirs":true,"Destination":"/private/var/folders/r6/346cfqyn76b_lx1nrcl5278c0000gp/T/NomadClient988174982/ed627e93-debe-834d-627b-28a11f3adea0/redis/secrets/file","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":420,"Source":"","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}"}],"Vault":{"Address":"http://127.0.0.1:8200","Enabled":true,"Grace":15000000000,"RenewToken":false,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":true,"Key":"","ServerName":"","Verify":true},"Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":13,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false},"Wait":{"Enabled":false,"Min":0,"Max":0}}
2019/01/29 09:18:49.045184 [INFO] (runner) creating watcher
2019/01/29 09:18:49.045257 [INFO] (runner) starting
2019/01/29 09:18:49.045267 [DEBUG] (runner) running initial templates
2019/01/29 09:18:49.045274 [INFO] (runner) initiating run
2019/01/29 09:18:49.045293 [DEBUG] (runner) checking template 7129828742b245e768317c62a6a47c91
2019/01/29 09:18:49.045570 [DEBUG] (runner) was not watching 1 dependencies
2019/01/29 09:18:49.045586 [DEBUG] (watcher) adding vault.read(secret/data/secret)
2019/01/29 09:18:49.045601 [DEBUG] (runner) diffing and updating dependencies
2019/01/29 09:18:49.045608 [DEBUG] (runner) watching 1 dependencies
2019/01/29 09:18:49.107490 [DEBUG] client: updated allocations at index 13 (total 1) (pulled 0) (filtered 1)
2019/01/29 09:18:49.107540 [DEBUG] client: allocs: (added 0) (removed 0) (updated 0) (ignore 1)
2019/01/29 09:18:50.023577 [DEBUG] http: Request GET /v1/evaluation/658d59a1-8b41-a682-8d4e-8124d2b93cdd (220.158µs)
2019/01/29 09:18:50.024883 [DEBUG] http: Request GET /v1/evaluation/658d59a1-8b41-a682-8d4e-8124d2b93cdd/allocations (205.416µs)
2019/01/29 09:18:52.061164 [DEBUG] client: event from "Template" for task "redis" in alloc "ed627e93-debe-834d-627b-28a11f3adea0": Missing: vault.read(secret/data/secret)
2019/01/29 09:18:52.306257 [DEBUG] client: updated allocations at index 14 (total 1) (pulled 0) (filtered 1)
2019/01/29 09:18:52.306302 [DEBUG] client: allocs: (added 0) (removed 0) (updated 0) (ignore 1)
2019/01/29 09:18:53.063018 [WARN] (view) vault.read(secret/data/secret): vault.read(secret/data/secret): Get http://127.0.0.1:8200/v1/secret/data/secret: dial tcp 127.0.0.1:8200: connect: connection refused (retry attempt 1 after "250ms")
2019/01/29 09:18:56.804466 [WARN] (view) vault.read(secret/data/secret): vault.read(secret/data/secret): Get http://127.0.0.1:8200/v1/secret/data/secret: dial tcp 127.0.0.1:8200: connect: connection refused (retry attempt 2 after "500ms")
2019/01/29 09:19:00.754742 [WARN] (view) vault.read(secret/data/secret): vault.read(secret/data/secret): Get http://127.0.0.1:8200/v1/secret/data/secret: dial tcp 127.0.0.1:8200: connect: connection refused (retry attempt 3 after "1s")
2019/01/29 09:19:05.631764 [WARN] (view) vault.read(secret/data/secret): vault.read(secret/data/secret): Get http://127.0.0.1:8200/v1/secret/data/secret: dial tcp 127.0.0.1:8200: connect: connection refused (retry attempt 4 after "2s")
2019/01/29 09:19:11.723907 [WARN] (view) vault.read(secret/data/secret): vault.read(secret/data/secret): Get http://127.0.0.1:8200/v1/secret/data/secret: dial tcp 127.0.0.1:8200: connect: connection refused (retry attempt 5 after "4s")
2019/01/29 09:19:19.694141 [WARN] (view) vault.read(secret/data/secret): vault.read(secret/data/secret): Get http://127.0.0.1:8200/v1/secret/data/secret: dial tcp 127.0.0.1:8200: connect: connection refused (retry attempt 6 after "8s")
2019/01/29 09:19:31.724748 [WARN] (view) vault.read(secret/data/secret): vault.read(secret/data/secret): Get http://127.0.0.1:8200/v1/secret/data/secret: dial tcp 127.0.0.1:8200: connect: connection refused (retry attempt 7 after "16s")
```

When running `nc -l 8200` I get the following request:

```
GET /v1/secret/data/secret HTTP/1.1
Host: 127.0.0.1:8200
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip
```

### Job file (if appropriate)

```
job "example-vault" {
datacenters = ["dc1"]

group "cache" {
task "redis" {
driver = "docker"

config {
image = "redis:3.2"
}

resources {
cpu = 500
memory = 256
}

template {
data = <

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.