hashicorp / hashicorp/consul-template

deduplication data can exceed consul value limit 512-kB

Open
#1,135 3 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Go
Stars
4.8k
Forks
801
Avg merge
4h 5m
Merged PRs (30d)
6

Description

### Consul Template version

```
$ consul-template -v
consul-template v0.19.5 (57b6c71)
```

### Configuration

```hcl
log_level = "warn"

max_stale = "10m"

consul {}

deduplicate {
enabled = true
prefix = "consul-template/my-proxy/dedup/"
}

template {
source = "/etc/consul-template.d/haproxy.conf"
destination = "/etc/haproxy/haproxy.cfg"

command = "/opt/consul/bin/consul lock -n=1 locks/fubar /usr/sbin/service haproxy reload"
command_timeout = "30s"

# limit restarts by waiting for new typeserver(s) to quiesce
wait {
min = "60s"
max = "90s"
}

# This allows us to mix consul-template with ansible template
left_delimiter = "[["
right_delimiter = "]]"
}
```

Template:

```liquid
[[/*
marshall the available service names
The service names we want look like "baz-*"
Ultimately we want to create a single backend
for all nodes with a matching service,
plus a backend for each matching service.
*/]]
[[- $mode := "fubar" -]]
[[- $pattern := "\\bfubar\\b" ]]
[[- range $i, $service := services -]]
[[- $serviceName := .Name -]]
[[- $matches := $serviceName | regexMatch "^baz-[^\\(]+$" -]]
[[- if $matches -]]
[[- range $j, $tag := .Tags -]]
[[ if $tag | regexMatch $pattern -]]
[[- $serviceRef := printf "%s.%s" $tag $serviceName -]]
[[/* identity map of tags, for backend-all */]]
[[- scratch.MapSet "all" $tag $tag -]]
[[/* map of version-specific backend names and tags.name ids */]]
[[- scratch.MapSet "services" $serviceName $serviceRef ]]
[[- end -]]
[[- end -]]
[[- else ]]
[[- end -]]
[[- end]]

frontend all
bind /var/run/my-proxy.sock
default_backend all

option http-buffer-request
declare capture request len 4000
http-request capture req.body id 0
capture cookie fubarhost= len 32

[[- range $name, $ref := scratch.Get "services" -]]
[[- $version := $name | regexReplaceAll "^baz-" "" ]]
acl is-[[$version]] hdr(X-Fubar-Version) -i [[$version]]
use_backend be-[[$version]] if is-[[$version]]
[[- end]]

# fallthrough, to handle bad version headers: must be last!
acl has-version hdr(X-Fubar-Version) -m found
use_backend be-bad-version if has-version

[[/*
write out all the backend servers, ignoring consul health status,
using `|any`
This way the server list should not change frequently,
even if the health check is flapping.
*/]]
backend all
[[- range $i, $tag := scratch.MapValues "all" -]]
[[- $servicePat := printf "%s.baz-all|any" $tag -]]
[[- range $j, $node := service $servicePat ]]
[[- $label := printf "all-%s" $node.Address | replaceAll "." "-" -]]
[[- $hostport := printf "%s:%d" $node.Address $node.Port -]]
[[- if scratch.Key $node.Address -]]
[[/* we have already seen this address */]]
[[- else -]]
[[- scratch.Set $hostport $hostport]]
server [[$label]] [[$hostport]] check port 8081 cookie [[$hostport]]
[[- end -]]
[[- end -]]
[[- end]]

[[ range $name, $ref := scratch.Get "services" -]]
[[- $servicePat := printf "%s|any" $ref -]]
[[- $serviceNodes := service $servicePat -]]
[[- $version := $name | regexReplaceAll "^baz-" "" ]]
backend be-[[$version]]
[[- if lt (len $serviceNodes) 1 ]]
mode http
errorfile 503 /etc/haproxy/errors/not-compatible.http
[[- else -]]
[[- range $j, $node := $serviceNodes -]]
[[- $track := printf "all-%s" $node.Address | replaceAll "." "-" -]]
[[- $label := printf "be-%s-%s" $version $node.Address | replaceAll "." "-" ]]
server [[$label]] [[$node.Address]]:[[$node.Port]] cookie [[$node.Address]]:[[$node.Port]] track all/[[$track]]
[[- end ]][[end]]
[[end]]
```

### Command

```shell
/usr/local/bin/consul-template -config /etc/consul-template.conf
```

### Debug output

https://gist.github.com/mblakele/7efefca93afe7b4e77cbbc886abc7b9c

### Expected behavior

What should have happened?

Ideally consul-template should run without errors. Maybe if the data is too large it could be broken up into multiple keys.

Another possible solution would be to ignore dedup if the data is too large, turning this error into a warning.

Failing that, it would be nice if consul-template gave me more details about what's wrong. I'd like to see what's in this chunk of data so I can try to reduce the size.

There's no old value, so I can't look at that. Looking at other values in the kv shows me that it's binary data, and I don't know how to decode or parse it.

```
$ /opt/consul/bin/consul kv get -detailed -base64 'consul-template/my-proxy/dedup/1442ff1543aaf681cfa72e1ef3401af6/data'
Error! No key exists at: consul-template/my-proxy/dedup/1442ff1543aaf681cfa72e1ef3401af6/data
```

Any ideas for resolving this problem?

Incidentally, is there any cleanup process? I seem to have about 250 of these dedup values in consul.

### Actual behavior

```
2018/08/31 21:45:19.605555 [ERR] (runner) failed to update dependency data for de-duplication: failed to write 'consul-template/my-proxy/dedup/1442ff1543aaf681cfa72e1ef3401af6/data': Unexpected response code: 413 (Value exceeds 524288 byte limit)
```

The consul docs explain that there's a 512-kB limit on values in the kv store: https://www.consul.io/docs/faq.html

### Steps to reproduce

Reproducing this probably requires a large cluster. We're seeing it in one environment that has 300 instances registered for various services.

### References

N/A

Contributor guide

Open the contributing guide

Research direction

No source file, test, or entry point is named. Trace the deduplication write that produces the reported Consul key, then reproduce it with data over Consul's 524288-byte limit. Done should follow an agreed behavior for oversized data, such as handling it safely or reporting actionable diagnostics, and clarify cleanup of old deduplication values.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.