Templates secrets are read from Vault when Vault integration isn't enabled
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
```
Nomad v0.8.7 (21a2d93eecf018ad2209a5eab6aae6c359267933)
```
Tested against latest master too
### Operating system and Environment details
macOS
### Issue
If a job task has a template stanza referencing Vault, Nomad would make a Vault call even if Nomad Vault integration is not configured. In such case, Vault would make unauthenticated API calls. These calls will fail (either through connection error, or permission error), and they will get retried multiple times.
### Reproduction steps
1. Start a nomad dev agent `nomad agent --dev` (or a non-dev agent too)
2. Submit job spec below `nomad job run ./example-vault.hcl`
3. Monitor nomad and alloc
I expect task to fail start immediately, but see many references to reading with exponential backoff. You can spin up `nc -l 8200` server to dump the request API.
### Nomad logs (if appropriate)
```
==> No configuration files loaded
==> Starting Nomad agent...
==> Nomad agent configuration:
Advertise Addrs: HTTP: 127.0.0.1:4646; RPC: 127.0.0.1:4647; Serf: 127.0.0.1:4648
Bind Addrs: HTTP: 127.0.0.1:4646; RPC: 127.0.0.1:4647; Serf: 127.0.0.1:4648
Client: true
Log Level: DEBUG
Region: global (DC: dc1)
Server: true
Version: 0.8.7
==> Nomad agent started! Log data will stream in below:
[...]
2019/01/29 09:18:49.032887 [DEBUG] client: starting task runners for alloc 'ed627e93-debe-834d-627b-28a11f3adea0'
2019/01/29 09:18:49.033004 [DEBUG] client.alloc_watcher: deadline for alloc "ed627e93-debe-834d-627b-28a11f3adea0" is at 2019-01-29 09:23:49.032972104 -0500 EST m=+306.896702646 (deploy=false checks=true)
2019/01/29 09:18:49.033060 [DEBUG] client: starting task context for 'redis' (alloc 'ed627e93-debe-834d-627b-28a11f3adea0')
2019/01/29 09:18:49.044822 [INFO] (runner) creating new runner (dry: false, once: false)
2019/01/29 09:18:49.045119 [DEBUG] (runner) final config: {"Consul":{"Address":"","Auth":{"Enabled":false,"Username":"","Password":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"","Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":13,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000},"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"WARN","MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0"},"Templates":[{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"supersecret = \"{{with secret \"secret/data/secret\"}}{{.Data.data.value}}{{end}}\"\n","CreateDestDirs":true,"Destination":"/private/var/folders/r6/346cfqyn76b_lx1nrcl5278c0000gp/T/NomadClient988174982/ed627e93-debe-834d-627b-28a11f3adea0/redis/secrets/file","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":420,"Source":"","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}"}],"Vault":{"Address":"http://127.0.0.1:8200","Enabled":true,"Grace":15000000000,"RenewToken":false,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":true,"Key":"","ServerName":"","Verify":true},"Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":13,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false},"Wait":{"Enabled":false,"Min":0,"Max":0}}
2019/01/29 09:18:49.045184 [INFO] (runner) creating watcher
2019/01/29 09:18:49.045257 [INFO] (runner) starting
2019/01/29 09:18:49.045267 [DEBUG] (runner) running initial templates
2019/01/29 09:18:49.045274 [INFO] (runner) initiating run
2019/01/29 09:18:49.045293 [DEBUG] (runner) checking template 7129828742b245e768317c62a6a47c91
2019/01/29 09:18:49.045570 [DEBUG] (runner) was not watching 1 dependencies
2019/01/29 09:18:49.045586 [DEBUG] (watcher) adding vault.read(secret/data/secret)
2019/01/29 09:18:49.045601 [DEBUG] (runner) diffing and updating dependencies
2019/01/29 09:18:49.045608 [DEBUG] (runner) watching 1 dependencies
2019/01/29 09:18:49.107490 [DEBUG] client: updated allocations at index 13 (total 1) (pulled 0) (filtered 1)
2019/01/29 09:18:49.107540 [DEBUG] client: allocs: (added 0) (removed 0) (updated 0) (ignore 1)
2019/01/29 09:18:50.023577 [DEBUG] http: Request GET /v1/evaluation/658d59a1-8b41-a682-8d4e-8124d2b93cdd (220.158µs)
2019/01/29 09:18:50.024883 [DEBUG] http: Request GET /v1/evaluation/658d59a1-8b41-a682-8d4e-8124d2b93cdd/allocations (205.416µs)
2019/01/29 09:18:52.061164 [DEBUG] client: event from "Template" for task "redis" in alloc "ed627e93-debe-834d-627b-28a11f3adea0": Missing: vault.read(secret/data/secret)
2019/01/29 09:18:52.306257 [DEBUG] client: updated allocations at index 14 (total 1) (pulled 0) (filtered 1)
2019/01/29 09:18:52.306302 [DEBUG] client: allocs: (added 0) (removed 0) (updated 0) (ignore 1)
2019/01/29 09:18:53.063018 [WARN] (view) vault.read(secret/data/secret): vault.read(secret/data/secret): Get http://127.0.0.1:8200/v1/secret/data/secret: dial tcp 127.0.0.1:8200: connect: connection refused (retry attempt 1 after "250ms")
2019/01/29 09:18:56.804466 [WARN] (view) vault.read(secret/data/secret): vault.read(secret/data/secret): Get http://127.0.0.1:8200/v1/secret/data/secret: dial tcp 127.0.0.1:8200: connect: connection refused (retry attempt 2 after "500ms")
2019/01/29 09:19:00.754742 [WARN] (view) vault.read(secret/data/secret): vault.read(secret/data/secret): Get http://127.0.0.1:8200/v1/secret/data/secret: dial tcp 127.0.0.1:8200: connect: connection refused (retry attempt 3 after "1s")
2019/01/29 09:19:05.631764 [WARN] (view) vault.read(secret/data/secret): vault.read(secret/data/secret): Get http://127.0.0.1:8200/v1/secret/data/secret: dial tcp 127.0.0.1:8200: connect: connection refused (retry attempt 4 after "2s")
2019/01/29 09:19:11.723907 [WARN] (view) vault.read(secret/data/secret): vault.read(secret/data/secret): Get http://127.0.0.1:8200/v1/secret/data/secret: dial tcp 127.0.0.1:8200: connect: connection refused (retry attempt 5 after "4s")
2019/01/29 09:19:19.694141 [WARN] (view) vault.read(secret/data/secret): vault.read(secret/data/secret): Get http://127.0.0.1:8200/v1/secret/data/secret: dial tcp 127.0.0.1:8200: connect: connection refused (retry attempt 6 after "8s")
2019/01/29 09:19:31.724748 [WARN] (view) vault.read(secret/data/secret): vault.read(secret/data/secret): Get http://127.0.0.1:8200/v1/secret/data/secret: dial tcp 127.0.0.1:8200: connect: connection refused (retry attempt 7 after "16s")
```
When running `nc -l 8200` I get the following request:
```
GET /v1/secret/data/secret HTTP/1.1
Host: 127.0.0.1:8200
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip
```
### Job file (if appropriate)
```
job "example-vault" {
datacenters = ["dc1"]
group "cache" {
task "redis" {
driver = "docker"
config {
image = "redis:3.2"
}
resources {
cpu = 500
memory = 256
}
template {
data = <
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.