hashicorp / hashicorp/consul-template

Consul Template dry is producing a different result than not-dry

Open
#1,204 4 comments 0 reactions 0 assignees View on GitHub
bug unreproducible
Dominant language
Go
Stars
4.8k
Forks
801
Avg merge
4h 5m
Merged PRs (30d)
6

Description

Please note that the Consul Template issue tracker is reserved
for bug reports and enhancements. For general usage questions,
please use the Consul mailing list:

https://groups.google.com/forum/#!forum/consul-tool

### Consul Template version

consul-template v0.19.5 (57b6c71)

### Configuration

(I have removed the commented out lines here for readability)

```hcl

consul {
address = "127.0.0.1:8500"
retry {

enabled = true
attempts = 0
backoff = "250ms"
max_backoff = "30s"
}
}

log_level = "info"

template { source = "/webApps/nginx/conf/env/stage/canary/canary_demo.tmpl" destination = "/webApps/nginx/conf/env/stage/canary/canary_demo.rendered" command = "/bin/sh /webApps/nginx/scripts/reloadNginx.sh" perms = 0777 backup = true wait { min = "2s" max = "10s" } }
template { source = "/webApps/nginx/conf/env/stage/upstreams/consul-upstream.tmpl" destination = "/webApps/nginx/conf/env/stage/upstreams/consul-upstream_conf" perms = 0777 backup = true wait { min = "2s" max = "10s" } }
template { source = "/webApps/nginx/conf/env/stage/locations/consul-locations.tmpl" destination = "/webApps/nginx/conf/env/stage/locations/consul-locations_conf" command = "/bin/sh /webApps/nginx/scripts/reloadNginx.sh" perms = 0777 backup = true wait { min = "2s" max = "10s" } }
```

(I'm only pasting the template that we are having issues with, which is canary_demo.tmpl)

```liquid
# Copy-paste your Consul Template template here

{{ define "canaryHostMap" }}
{{- $appName := . }}
{{- $varSafeAppName := $appName | regexReplaceAll "[^a-zA-Z0-9_]" "_" }}
{{- if keyExists (printf "canary/apps/%s/groupingStrategy" $appName) }}
{{- with $appDetails := tree (printf "canary/apps/%s/" $appName) | explode }}
# appName: {{ $appName }}
# groupingStrategy: {{ $groupingStrategy := index $appDetails "groupingStrategy" }}{{ $groupingStrategy }}
# scaleMode: {{ index $appDetails "scaleMode" }}
map {{ if eq $groupingStrategy "random_user" }}$cookie_aid_3 {{ else }} {{ if eq $groupingStrategy "random_pid" }}$cookie_pid {{ else }}$http_x_correlation_id {{ end }}{{ end }} ${{ $varSafeAppName }}_host {
{{ range service $appName }}
{{- $id := .ID }}
{{- $host := .Address }}
{{- with $versions := index $appDetails "versions" }}
{{- with $version := index $versions $id }}
{{- $groups := index $version "groups" }}
{{- $weight := index $version "weight" }}
{{- $isFeatureCandidate := index $version "feature_candidate" }}
# id: {{ $id }}
# traffic %: {{ $weight }}
# feature_candidate: {{ $isFeatureCandidate }}
{{ range ($groups | split ",") }}{{ . }} {{ $host }};
{{ end }}
{{ end }}
{{ end -}}
{{ else }}# no versions for app{{ end }}
} {{/* end nginx map directive */}}
{{ end }}
{{ else }}
# appName: {{ $appName }}
# no registered services for app
map $http_x_correlation_id ${{ $varSafeAppName }}_host {}
{{ end }}
{{ end }}
{{/* End 'canaryHostMap' definition */}}

{{/*
---------- Everything above this line will be seperated into it's own file ----------
---------- Below is what your canary config will look like ----------
*/}}

{{/* Canary host variable generation */}}
{{ executeTemplate "canaryHostMap" "locator-stage" }}
{{ executeTemplate "canaryHostMap" "item-detail-services-stage" }}
```

### Command

```shell
consul-template -template "/webApps/nginx/conf/env/$PLATFORM_ENVIRONMENT/canary/canary_demo.tmpl:/webApps/nginx/conf/env/$PLATFORM_ENVIRONMENT/canary/canary_demo.rendered" -once

consul-template -template "/webApps/nginx/conf/env/$PLATFORM_ENVIRONMENT/canary/canary_demo.tmpl:/webApps/nginx/conf/env/$PLATFORM_ENVIRONMENT/canary/canary_demo.rendered" -once -dry
```

### Debug output

I think this is the important information

```
2019/04/05 15:10:42.302273 [DEBUG] (runner) initiating run
2019/04/05 15:10:42.302279 [DEBUG] (runner) checking template 456dec88739dd18c1cbfeecd0585ce56
2019/04/05 15:10:42.302695 [DEBUG] (runner) checking template 13a2ab9ad84269b26f8a668350fd197a
2019/04/05 15:10:42.303096 [DEBUG] (runner) checking template 0ab072388fd1c358ed325191ff5c70e4
2019/04/05 15:10:42.303555 [DEBUG] (runner) rendering "/webApps/nginx/conf/env/stage/locations/consul-locations.tmpl" => "/webApps/nginx/conf/env/stage/locations/consul-locations_conf"
2019/04/05 15:10:42.303607 [DEBUG] (runner) diffing and updating dependencies
2019/04/05 15:10:42.303625 [DEBUG] (runner) kv.list(canary/apps/locator-stage/) is still needed
2019/04/05 15:10:42.303632 [DEBUG] (runner) kv.get(canary/apps/item-detail-services-stage/groupingStrategy) is still needed
2019/04/05 15:10:42.303640 [DEBUG] (runner) catalog.services is still needed
2019/04/05 15:10:42.303646 [DEBUG] (runner) kv.get(picking/location) is still needed
2019/04/05 15:10:42.303662 [DEBUG] (runner) health.service(picking|passing) is still needed
2019/04/05 15:10:42.303678 [DEBUG] (runner) kv.get(ecsb/foundation) is still needed
2019/04/05 15:10:42.303690 [DEBUG] (runner) kv.get(picking/stage/external-uri) is still needed
2019/04/05 15:10:42.303703 [DEBUG] (runner) kv.get(canary/apps/locator-stage/groupingStrategy) is still needed
2019/04/05 15:10:42.303720 [DEBUG] (runner) enabling template-specific quiescence for "0ab072388fd1c358ed325191ff5c70e4"
2019/04/05 15:10:42.303740 [DEBUG] (runner) watching 8 dependencies
```

What's weird is that the KV values it says "is still needed" exist in the Consul Server KV. For example:

```
consul kv get canary/apps/locator-stage/groupingStrategy
random_session
```

### Expected behavior

The output from the dry command should match the output to the rendered file

### Actual behavior

Dry output:
```
      # appName: locator-stage

      # groupingStrategy: random_session

      # scaleMode: linear

      map  $http_x_correlation_id  $locator_stage_host {

              # id:                locator-stage-2-37-3

              # traffic %:         1

              # feature_candidate: false

              default locator-stage-2-37-3.kroger.com;

      }

    # appName: item-detail-services-stage

    # no registered services for app

    map $http_x_correlation_id $item_detail_services_stage_host {}
```

File output:
```
# appName: locator-stage
# groupingStrategy: random_session
# scaleMode: linear
map $http_x_correlation_id $locator_stage_host {
# no versions for app
}

# appName: item-detail-services-stage
# no registered services for app
map $http_x_correlation_id $item_detail_services_stage_host {}
```

### Steps to reproduce

1. run `consul-template -template "/webApps/nginx/conf/env/$PLATFORM_ENVIRONMENT/canary/canary_demo.tmpl:/webApps/nginx/conf/env/$PLATFORM_ENVIRONMENT/canary/canary_demo.rendered" -once -dry`
2. run `consul-template -template "/webApps/nginx/conf/env/$PLATFORM_ENVIRONMENT/canary/canary_demo.tmpl:/webApps/nginx/conf/env/$PLATFORM_ENVIRONMENT/canary/canary_demo.rendered" -once`

### References

Are there any other GitHub issues (open or closed) that should
be linked here? For example:
- GH-1234
- ...

N\A

Contributor guide

Open the contributing guide

Research direction

Reproduce the mismatch with the two consul-template commands in the issue, using the supplied HCL configuration and canary_demo.tmpl template. Compare the dry output with the rendered file output, then trace the dry-run rendering path and its handling of the shown Consul KV and service dependencies. Done means both commands produce matching output for this reproduction.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.