Warn on missing dependencies or data blocking template rendering

TL;DR: It would be cool if templates that are waiting on blocking queries had a warning printed after some amount of time indicating what they're waiting on. In our case, a missing Consul key blocked the rendering of a (complicated) template, which resulted in some head scratching until we hunted down the missing key.

Consul Template version

$ consul-template -v
 v0.26.0 (3b7f233a)

Configuration

template {
  source        = "/tmp/ct-test/test.ctmpl"
  destination   = "/tmp/ct-test/out.txt"
}

{{ key "missingkey" }}

(Any key not in the K/V store will suffice, missingkey is just an example)

Command

$ consul agent -dev # in another shell, easy-to-produce empty K/V store
$ consul-template -config /tmp/ct-test/config.hcl -log-level=debug

Debug output

On startup:

https://gist.github.com/rwiggins/29d1d7da822e16bbf482613a6f9d2c5d

then once the key is added with $ consul kv put testkey 1:

https://gist.github.com/rwiggins/29d1d7da822e16bbf482613a6f9d2c5d#gistcomment-3806914

Actual behavior

Rendering of the template is blocked until the key is added to Consul, specifically without any log messages indicating that (except at debug level).

In other words, Consul Template is executed, and it appears "hung" to the user — no logs, no warnings, just no template, until an investigation is made. In a simple template like the above, it's not really a mystery... however, this issue took a bit of hunting recently in one of our test builds, which has a substantially more complicated template.

Expected behavior

I understand that the blocking {{ key }} query is both desired and documented. However, I guess my hope would be a warning message after some time, under the assumption that it's atypical for blocking queries to take more than e.g. 5 seconds. Maybe if a template is blocked because of missing dependencies for N seconds, produce a warning message that enumerates the missing data? For example,

[WARN] (runner) template "/tmp/ct-test/test.ctmpl" rendering waiting on missing dependencies: kv.block(testkey), kv.block(testkey2)

I am not entirely sure how long to wait before warning. Maybe a second or two or five?

The example above only shows missing Consul keys, but I'm sure the same could apply to all sorts of other data sources. (See 2nd referenced issue below, for example.)

This type of warning message would have turned a recent puzzled investigation into an immediate resolution, at least in our use of Consul Template.

References

• https://github.com/hashicorp/consul-template/issues/771
• https://github.com/hashicorp/consul-template/issues/336

Reproduce the issue with the shown Consul Template configuration, a missing key, and debug logging, then inspect the template-rendering and blocking-query paths described in the report. Done means a non-debug warning appears after a configurable or agreed delay and identifies the missing dependencies without changing the intentional blocking behavior.