hashicorp / hashicorp/consul-template

Allow 'file' retries if file doesn't exist yet

Open
#1,366 3 comments 5 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
4.8k
Forks
801
Avg merge
4h 5m
Merged PRs (30d)
6

Description

The 'file' function will cause consul-template to immediately exit if the file in question doesn't exist, but it would be super useful if it supported retries like every other function. We have 2 containers, both run consul-template but the first reads secrets from Vault and renders a JSON file and the second reads the JSON file via a common volume. We don't want the second container to also read from Vault because it is a generic sidecar and the assumption the JSON file could come from anywhere and it is up to the first container to provide it. The problem is the second container isn't guaranteed to start after the first so there is no guarantee the JSON file exists yet. If there were some retries then the second container could just wait a few seconds for it to become available then render the template.

### Consul Template version

consul-template v0.24.1 (c54d0abc)

### Configuration

/tmp/config.hcl
```hcl
log_level = "debug"
template {
source = "/tmp/oidc.conf.ctmpl"
destination = "/tmp/oidc.conf"
}
```

/tmp/oidc.conf.ctmpl
```liquid
{{ $oidc := file "/tmp/oidc-missing.json" | parseJSON -}}
provider = "oidc"
client_id = "{{ $oidc.id }}"
client_secret = "{{ $oidc.secret }}"
```

/tmp/oidc.json
```liquid
{
"id" : "foo",
"secret" : "bar"
}
```

### Command

```shell
consul-template -config /tmp/config.hcl -once
```

### Expected behavior

What we are hoping consul-template retries using standard exponential backoff until you run `cp /tmp/oid.json /tmp/oidc-missing.json` so it can finish rendering.

### Actual behavior

Consul-template exits immediately with:

```
2020/04/14 23:00:55.845820 [INFO] consul-template v0.24.1 (c54d0abc)
2020/04/14 23:00:55.845858 [INFO] (runner) creating new runner (dry: false, once: true)
2020/04/14 23:00:55.846330 [INFO] (runner) creating watcher
2020/04/14 23:00:55.846478 [INFO] (runner) starting
2020/04/14 23:00:55.847644 [ERR] (view) file(/tmp/oidc-missing.json): stat /tmp/oidc-missing.json: no such file or directory (exceeded maximum retries)
2020/04/14 23:00:55.847654 [ERR] (runner) watcher reported error: file(/tmp/oidc-missing.json): stat /tmp/oidc-missing.json: no such file or directory
2020/04/14 23:00:55.851655 [ERR] (cli) file(/tmp/oidc-missing.json): stat /tmp/oidc-missing.json: no such file or directory
```

### Steps to reproduce

1. Create the files mentioned in the Configuration section above
2. run `consul-template -config /tmp/config.hcl -once`

Contributor guide

Open the contributing guide

Research direction

Start with the template's file function and the watcher behavior exercised by `consul-template -config /tmp/config.hcl -once`. Reproduce the missing-file case using the configuration and template shown, then define completion as retrying with backoff until `/tmp/oidc-missing.json` appears and rendering succeeds without an immediate exit.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.