hashicorp / hashicorp/consul-template
dependencies DDoS warning misleading and overly spammy
- Dominant language
- Go
- Stars
- 4.8k
- Forks
- 801
- Avg merge
- 4h 5m
- Merged PRs (30d)
- 6
Description
In my particular configuration, I see a lot of these messages generated by consul-template, one generated every few seconds which seems far more verbose than necessary:
```
consul-template[941]: 2020/08/19 02:54:31.389950 [WARN] (runner) watching 141 dependencies - watching this many dependencies could DDoS your consul cluster
consul-template[941]: 2020/08/19 02:54:33.018359 [WARN] (runner) watching 141 dependencies - watching this many dependencies could DDoS your consul cluster
consul-template[941]: 2020/08/19 02:54:37.195036 [WARN] (runner) watching 141 dependencies - watching this many dependencies could DDoS your consul cluster
consul-template[941]: 2020/08/19 02:54:46.384132 [WARN] (runner) watching 141 dependencies - watching this many dependencies could DDoS your consul cluster
consul-template[941]: 2020/08/19 02:54:47.028610 [WARN] (runner) watching 141 dependencies - watching this many dependencies could DDoS your consul cluster
consul-template[941]: 2020/08/19 02:54:48.284239 [WARN] (runner) watching 141 dependencies - watching this many dependencies could DDoS your consul cluster
consul-template[941]: 2020/08/19 02:54:48.814016 [WARN] (runner) watching 141 dependencies - watching this many dependencies could DDoS your consul cluster
consul-template[941]: 2020/08/19 02:54:51.576816 [WARN] (runner) watching 141 dependencies - watching this many dependencies could DDoS your consul cluster
consul-template[941]: 2020/08/19 02:54:53.775854 [WARN] (runner) watching 141 dependencies - watching this many dependencies could DDoS your consul cluster
consul-template[941]: 2020/08/19 02:54:54.278439 [WARN] (runner) watching 141 dependencies - watching this many dependencies could DDoS your consul cluster
consul-template[941]: 2020/08/19 02:55:05.046358 [WARN] (runner) watching 141 dependencies - watching this many dependencies could DDoS your consul cluster
```
An option to suppress these messages entirely, or at least reduce the rate to something more sane would make sense - perhaps to once every time the template is rendered would be a good option (in the above log, no template renders happened, it is apparent when a template render happens as the command defined in the template{} blocks writes to stdout which appears in the log).
Furthermore, while investigating what the dependencies were to see if any efficiencies could be made, I discovered that consul-template was already intelligently using a keyprefix watch against Consul meaning that only a few Consul dependencies were actually being watched, the remaining 130-odd dependencies were actually all Vault KV secrets! Needless to say the warning about DDoSing my consul cluster is definitely misleading since it's my Vault cluster I would have to worry about.
My understanding is normally Vault secrets would use the lease time (or for certificates the certificate expiry, at least in newer versions of consul-template, when a lease isn't generated) to determine when it should recheck, but for the Vault KV secrets engine neither option would work. I can only assume consul-template will poll Vault in such cases on some interval. I couldn't see anywhere in the docs describing the interval at which consul-template would poll Vault for changes, nor any configuration options to control the poll interval so it's difficult to judge what too high constitutes here and potential mitigating options (my understanding is dedupe mode wouldn't help here).
Contributor guide
Research direction
Start at the runner warning shown in the issue and trace how dependency counts and polling events trigger it. Compare Consul and Vault KV dependencies, including the reported polling behavior, and identify what tests cover these warnings. Done means the warning is accurate about the affected service and is not emitted repeatedly without a meaningful change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100