hashicorp / hashicorp/consul-template

Non-interleaved template write and command exec

Open
#1,672 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
4.8k
Forks
801
Avg merge
4h 5m
Merged PRs (30d)
6

Description

## Consul Template version

`consul-template v0.29.5 (f07ce88)`

## Issue

During the course of a template update run, daemons repeatedly reload mismatched TLS materials, until only at the end of the run when each and every file has been updated, the TLS materials become fully consistent again. This occurs because daemon refreshes are interleaved with the updating of individual files, files that only make sense when interpreted as an atomic unit.

## Discussion

This issue came about after finishing the mTLS tutorials for [Nomad](https://developer.hashicorp.com/nomad/tutorials/integrate-vault/vault-pki-nomad) and [Consul](https://developer.hashicorp.com/consul/tutorials/vault-secure/vault-pki-consul-secure-tls). `consul-template` renders templates and outputs updated files. The key here is that each template file update is immediately followed by its corresponding exec command. In our case, we leverage this system to ask the Consul and Nomad daemons to reload their TLS configurations in response to expiring TTLs, as outlined in the tutorials.

Take the case of a pair of files that represent a TLS certificate and private key. Before running, the daemon has cert A and corresponding private key A. When `consul-template` runs the first `template` stanza (let's choose the `cert` as our example) then cert is updated to cert B while private key remains at A. `consul-template` obediently executes the daemon reload command, the daemon responds, and is surprised to find mismatched private key A and certificate B.

Logs indicate that at first mismatched update the cluster is prone to spasming for a while, refusing their own certificates and each others', breaking cluster communication, and being generally unusable, partially nullifying their value. Meanwhile, back at the barn, `consul-template` dutifully continues its cycle of updating then reloading, updating the reloading until all on-disk files are back in congruence, and the daemons have been force-reloaded N times to their & our detriment.

## Ideal

Ideally, the nomad and consul daemons would be notified only once each, after all TLS files were updated on disk.

As an idea, `consul-template` structures its aggregate `template` stanzas as a two-phase process.

*Phase 1*: All templates are rendered as warranted.

*Phase 2*: All exec commands associated with template output file updates are run. Identical commands are coalesced into a single invocation.

So instead of interleaving writes with exec commands, writes are performed first, then commands are exec'ed.

In terms of a solution, it seems we can live with the following ... shortcuts:
- Instead of the phases, adding a setting to delay invocation of the command exec in each `template` stanza. At this time, I would elect a (safe?) delay of say 1 minute.
- If `consul-template` can continue with remaining operations while waiting for earlier commands to finish, we can cause a delay by `sleep`ing before reloading the daemon in the exec command.
- For determining whether to coalesce commands, simple string comparison of the command exec strings is fine.

## Configuration

Even though this issue applies broadly, let's zoom in on Nomad server:

```hcl
template {
source = "templates/nomad-server-ca.crt.tpl"
destination = "/opt/nomad-server/tls/nomad-ca.crt"
exec {
command = "sh -c 'date && sudo /usr/bin/systemctl reload nomad-server'"
}
}

template {
source = "templates/nomad-server.crt.tpl"
destination = "/opt/nomad-server/tls/nomad.crt"
exec {
command = "sh -c 'date && sudo /usr/bin/systemctl reload nomad-server'"
}
}

template {
source = "templates/nomad-server.key.tpl"
destination = "/opt/nomad-server/tls/nomad.key"
exec {
command = "sh -c 'date && sudo /usr/bin/systemctl reload nomad-server'"
}
}
```

3 stanzas, 3 different files (CA cert, cert, key) all for configuring TLS on a nomad server instance, but in essence only 1 reload command. That command aught to be executed only after all updates are made to disk.

I'm curious what Hashicorp and other orgs are doing on their production instances?

Contributor guide

Open the contributing guide

Research direction

The issue names no source files, tests, or entry points. Start by locating the template rendering and exec scheduling paths, then determine how multiple template updates are coordinated. Done means files are updated before associated commands run and identical commands are invoked only once.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.