hashicorp / hashicorp/consul-template
Provide a config arg to toggle Termination on Error behavior
- Dominant language
- Go
- Stars
- 4.8k
- Forks
- 801
- Avg merge
- 4h 5m
- Merged PRs (30d)
- 6
Description
Regarding "Termination on Error", TFM states:
> By default Consul Template is highly fault-tolerant. If Consul is unreachable or a template changes, Consul Template will happily continue running. The only exception to this rule is if the optional command exits non-zero. In this case, Consul Template will also exit non-zero. The reason for this decision is so the user can easily configure something like Upstart or God to manage Consul Template as a service.
While that might work for some deployments, we need the exact opposite: `consul-template` can log the problem and then must must must continue operating. No exiting, no giving up, no spontaneous death. Never, ever quitting.
As background, what happened in the most recent affair is:
1. Nomad re-evaluated a job. One of the daemons targeted by our consul-template script was down during the re-evaluation.
2. At this most un-opportune moment, `consul-template` dutifully wrote out the rendered template and then executed the associated command.
3. Associated `command` attempts to HUP the daemon to reload its updated config files.
4. Daemon is down (on account of Nomad re-evaluation), so the HUP fails and the execution of `command` returns non-zero exit code.
5. `consul-template` responds in accordance with this Termination on Error stance and exits itself.
6. MTLS certificates are now no longer being updated on the node. All services are stuck holding TLS certs that are now becoming more stale by the moment.
7. Suddenly, for example, now Consul and Nomad clients on the node suddenly refuses to connect to anything and appears lost to the cluster, not to mention the myriad other services running on the node.
8. ZOMG what is this disruption our SLAs are sinking ... you can imagine the nth-order consequences cascading from this series of comical misfortunes.
I'd like to rewrite that story, focusing on the Termination on Error.
Given the two opposing needs, allowing supervisors to manage consul-template as a service, plus our needs (and a great many like us, I bet) to have consul-template as stable as possible, given its critical role in infra, I propose a boolean toggle as a new configuration option. This new option allows us to tailor `consul-template`'s behavior when a command exits non-zero. For the sake of backward compatibility, the default value preserves current behavior, but TFM screams & shouts & alerts us to the existence of this option and the perils of ignoring it. If you would allow me to have a go at writing the documentation specifically for the [new option](https://github.com/hashicorp/consul-template/blob/main/docs/configuration.md):
```
# This controls whether Consul Template exits non-zero whenever an optional command exits non-zero.
# For managing Consul Template under a supervisor like systemd, Upstart, or God, consider setting this to true.
# To have Consul Template continue operating even when a command fails, set this to false.
terminate_on_error = true
```
I added the recommended `your_command_here || true` to every single `command` block. After shooting ourselves in the foot, we remember this workaround at this point in time. Will we forever remember to avoid this footgun when writing scripts in future? Not guaranteed. And IMHO it's merely unnecessary cognitive overhead to remember with markedly undesirable consequences, a punishing price to pay for forgetting or overlooking it. As for the question of why not just restart consul-template from the supervisor? That would only lead to flapping. And monitoring is merely one factor, by definition after-the-fact. The option to toggle this terminating behavior off is during-the-fact and systemically wise, and will restore the warm, fuzzy feelings we want to keep associating with this tool.
Thanks!
Contributor guide
Assessment
This issue has not been assessed yet.