hashicorp / hashicorp/consul

There appears to be an undocumented hard limit for consul blocking queries targeting KV

Open
#8,995 0 comments 0 reactions 0 assignees View on GitHub
theme/kv theme/performance
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue
We deployed a new feature to our lower environments last week that would enable our SaaS fleet to grab runtime config from consul kv using the blocking api. We experienced some issues around resource constraints with this approach but after (seemingly) clearing those up, I began loadtesting an identical consul cluster with a simple script to generate _n_ watches. I noticed that the blocking api becomes unresponsive somewhere between 9000-10000 _n_. This loadtesting doesn't seem to stress the consul cluster at all so I feel that we should be safe to increase the blocking query limit with some configuration flag that may or may not exist in consul documentation.

#### Reproduction Steps

Try the following script (dependency: https://www.npmjs.com/package/consul) on a machine that can reach consul kv:
```
const consul = require('consul')({
host: 'some_consul_host',
port: 443,
secure: true,
promisify: true
})

const watch = () => {
consul
.watch({
method: consul.kv.get,
options: { key: 'some_key', recurse: true }
})
.on('change', function(data, res) {
console.log('data:', data)
checkDone()
})
.on('error', function(err) {
console.log('error:', err)
})
}

const N = 100
for (let i = 0; i < N; i++) {
watch()
}

let i = 0
const checkDone = () => {
if (++i === N) {
console.log('done')
process.exit(0)
}
}
```

This was tested against a 3 node consul cluster deployed with the official helm chart and the following resource constraints:
```
server:
replicas: 3
bootstrapExpect: 3
storage: 50Gi
resources:
requests:
memory: "4Gi"
cpu: "750m"
limits:
memory: "4Gi"
cpu: "750m"
```
### Consul info for both Client and Server

Client info

```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 12b16df3
version = 1.8.4
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = false
leader_addr = redacted
server = true
raft:
applied_index = 288315
commit_index = 288315
fsm_pending = 0
last_contact = 22.356471ms
last_log_index = 288315
last_log_term = 2
last_snapshot_index = 278835
last_snapshot_term = 2
latest_configuration = [{Suffrage:Voter ID:4d172f9b-350e-bb2c-ed0e-5563e2a4e913 Address:redacted:8300} {Suffrage:Voter ID:7e4db50f-d856-7fda-207d-f69c98544786 Address:redacted:8300}]
latest_configuration_index = 0
num_peers = 1
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 2
runtime:
arch = amd64
cpu_count = 2
goroutines = 82
max_procs = 2
os = linux
version = go1.14.6
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 3
members = 2
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 2
members = 2
query_queue = 0
query_time = 1
```

Server info

This would be difficult to obtain based on our networking standards but I can try to get it if pressed. Servers were deployed with latest version of consul helm chart

### Operating system and Environment details

Amazon EKS 1.15+

Contributor guide

Open the contributing guide

Research direction

No repository files or tests are named. Start by running the provided consul.kv.get watch script against a comparable cluster and characterize the behavior as the watch count approaches 9,000–10,000. Done means determining whether a hard limit exists and documenting the applicable configuration or limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, javascript, kubernetes
Domain
api, backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.