hashicorp / hashicorp/consul-template
Allow 'file' retries if file doesn't exist yet
- Dominant language
- Go
- Stars
- 4.8k
- Forks
- 801
- Avg merge
- 4h 5m
- Merged PRs (30d)
- 6
Description
The 'file' function will cause consul-template to immediately exit if the file in question doesn't exist, but it would be super useful if it supported retries like every other function. We have 2 containers, both run consul-template but the first reads secrets from Vault and renders a JSON file and the second reads the JSON file via a common volume. We don't want the second container to also read from Vault because it is a generic sidecar and the assumption the JSON file could come from anywhere and it is up to the first container to provide it. The problem is the second container isn't guaranteed to start after the first so there is no guarantee the JSON file exists yet. If there were some retries then the second container could just wait a few seconds for it to become available then render the template.
### Consul Template version
consul-template v0.24.1 (c54d0abc)
### Configuration
/tmp/config.hcl
```hcl
log_level = "debug"
template {
source = "/tmp/oidc.conf.ctmpl"
destination = "/tmp/oidc.conf"
}
```
/tmp/oidc.conf.ctmpl
```liquid
{{ $oidc := file "/tmp/oidc-missing.json" | parseJSON -}}
provider = "oidc"
client_id = "{{ $oidc.id }}"
client_secret = "{{ $oidc.secret }}"
```
/tmp/oidc.json
```liquid
{
"id" : "foo",
"secret" : "bar"
}
```
### Command
```shell
consul-template -config /tmp/config.hcl -once
```
### Expected behavior
What we are hoping consul-template retries using standard exponential backoff until you run `cp /tmp/oid.json /tmp/oidc-missing.json` so it can finish rendering.
### Actual behavior
Consul-template exits immediately with:
```
2020/04/14 23:00:55.845820 [INFO] consul-template v0.24.1 (c54d0abc)
2020/04/14 23:00:55.845858 [INFO] (runner) creating new runner (dry: false, once: true)
2020/04/14 23:00:55.846330 [INFO] (runner) creating watcher
2020/04/14 23:00:55.846478 [INFO] (runner) starting
2020/04/14 23:00:55.847644 [ERR] (view) file(/tmp/oidc-missing.json): stat /tmp/oidc-missing.json: no such file or directory (exceeded maximum retries)
2020/04/14 23:00:55.847654 [ERR] (runner) watcher reported error: file(/tmp/oidc-missing.json): stat /tmp/oidc-missing.json: no such file or directory
2020/04/14 23:00:55.851655 [ERR] (cli) file(/tmp/oidc-missing.json): stat /tmp/oidc-missing.json: no such file or directory
```
### Steps to reproduce
1. Create the files mentioned in the Configuration section above
2. run `consul-template -config /tmp/config.hcl -once`
Contributor guide
Research direction
Start with the template's file function and the watcher behavior exercised by `consul-template -config /tmp/config.hcl -once`. Reproduce the missing-file case using the configuration and template shown, then define completion as retrying with backoff until `/tmp/oidc-missing.json` appears and rendering succeeds without an immediate exit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cli
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100