hashicorp / hashicorp/consul-template

dependencies DDoS warning misleading and overly spammy

Open
#1,407 4 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
4.8k
Forks
801
Avg merge
4h 5m
Merged PRs (30d)
6

Description

In my particular configuration, I see a lot of these messages generated by consul-template, one generated every few seconds which seems far more verbose than necessary:

```
consul-template[941]: 2020/08/19 02:54:31.389950 [WARN] (runner) watching 141 dependencies - watching this many dependencies could DDoS your consul cluster
consul-template[941]: 2020/08/19 02:54:33.018359 [WARN] (runner) watching 141 dependencies - watching this many dependencies could DDoS your consul cluster
consul-template[941]: 2020/08/19 02:54:37.195036 [WARN] (runner) watching 141 dependencies - watching this many dependencies could DDoS your consul cluster
consul-template[941]: 2020/08/19 02:54:46.384132 [WARN] (runner) watching 141 dependencies - watching this many dependencies could DDoS your consul cluster
consul-template[941]: 2020/08/19 02:54:47.028610 [WARN] (runner) watching 141 dependencies - watching this many dependencies could DDoS your consul cluster
consul-template[941]: 2020/08/19 02:54:48.284239 [WARN] (runner) watching 141 dependencies - watching this many dependencies could DDoS your consul cluster
consul-template[941]: 2020/08/19 02:54:48.814016 [WARN] (runner) watching 141 dependencies - watching this many dependencies could DDoS your consul cluster
consul-template[941]: 2020/08/19 02:54:51.576816 [WARN] (runner) watching 141 dependencies - watching this many dependencies could DDoS your consul cluster
consul-template[941]: 2020/08/19 02:54:53.775854 [WARN] (runner) watching 141 dependencies - watching this many dependencies could DDoS your consul cluster
consul-template[941]: 2020/08/19 02:54:54.278439 [WARN] (runner) watching 141 dependencies - watching this many dependencies could DDoS your consul cluster
consul-template[941]: 2020/08/19 02:55:05.046358 [WARN] (runner) watching 141 dependencies - watching this many dependencies could DDoS your consul cluster
```

An option to suppress these messages entirely, or at least reduce the rate to something more sane would make sense - perhaps to once every time the template is rendered would be a good option (in the above log, no template renders happened, it is apparent when a template render happens as the command defined in the template{} blocks writes to stdout which appears in the log).

Furthermore, while investigating what the dependencies were to see if any efficiencies could be made, I discovered that consul-template was already intelligently using a keyprefix watch against Consul meaning that only a few Consul dependencies were actually being watched, the remaining 130-odd dependencies were actually all Vault KV secrets! Needless to say the warning about DDoSing my consul cluster is definitely misleading since it's my Vault cluster I would have to worry about.

My understanding is normally Vault secrets would use the lease time (or for certificates the certificate expiry, at least in newer versions of consul-template, when a lease isn't generated) to determine when it should recheck, but for the Vault KV secrets engine neither option would work. I can only assume consul-template will poll Vault in such cases on some interval. I couldn't see anywhere in the docs describing the interval at which consul-template would poll Vault for changes, nor any configuration options to control the poll interval so it's difficult to judge what too high constitutes here and potential mitigating options (my understanding is dedupe mode wouldn't help here).

Contributor guide

Open the contributing guide

Research direction

Start at the runner warning shown in the issue and trace how dependency counts and polling events trigger it. Compare Consul and Vault KV dependencies, including the reported polling behavior, and identify what tests cover these warnings. Done means the warning is accurate about the affected service and is not emitted repeatedly without a meaningful change.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.