There appears to be an undocumented hard limit for consul blocking queries targeting KV
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Overview of the Issue
We deployed a new feature to our lower environments last week that would enable our SaaS fleet to grab runtime config from consul kv using the blocking api. We experienced some issues around resource constraints with this approach but after (seemingly) clearing those up, I began loadtesting an identical consul cluster with a simple script to generate _n_ watches. I noticed that the blocking api becomes unresponsive somewhere between 9000-10000 _n_. This loadtesting doesn't seem to stress the consul cluster at all so I feel that we should be safe to increase the blocking query limit with some configuration flag that may or may not exist in consul documentation.
#### Reproduction Steps
Try the following script (dependency: https://www.npmjs.com/package/consul) on a machine that can reach consul kv:
```
const consul = require('consul')({
host: 'some_consul_host',
port: 443,
secure: true,
promisify: true
})
const watch = () => {
consul
.watch({
method: consul.kv.get,
options: { key: 'some_key', recurse: true }
})
.on('change', function(data, res) {
console.log('data:', data)
checkDone()
})
.on('error', function(err) {
console.log('error:', err)
})
}
const N = 100
for (let i = 0; i < N; i++) {
watch()
}
let i = 0
const checkDone = () => {
if (++i === N) {
console.log('done')
process.exit(0)
}
}
```
This was tested against a 3 node consul cluster deployed with the official helm chart and the following resource constraints:
```
server:
replicas: 3
bootstrapExpect: 3
storage: 50Gi
resources:
requests:
memory: "4Gi"
cpu: "750m"
limits:
memory: "4Gi"
cpu: "750m"
```
### Consul info for both Client and Server
Client info
```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 12b16df3
version = 1.8.4
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = false
leader_addr = redacted
server = true
raft:
applied_index = 288315
commit_index = 288315
fsm_pending = 0
last_contact = 22.356471ms
last_log_index = 288315
last_log_term = 2
last_snapshot_index = 278835
last_snapshot_term = 2
latest_configuration = [{Suffrage:Voter ID:4d172f9b-350e-bb2c-ed0e-5563e2a4e913 Address:redacted:8300} {Suffrage:Voter ID:7e4db50f-d856-7fda-207d-f69c98544786 Address:redacted:8300}]
latest_configuration_index = 0
num_peers = 1
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 2
runtime:
arch = amd64
cpu_count = 2
goroutines = 82
max_procs = 2
os = linux
version = go1.14.6
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 3
members = 2
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 2
members = 2
query_queue = 0
query_time = 1
```
Server info
This would be difficult to obtain based on our networking standards but I can try to get it if pressed. Servers were deployed with latest version of consul helm chart
### Operating system and Environment details
Amazon EKS 1.15+
Contributor guide
Research direction
No repository files or tests are named. Start by running the provided consul.kv.get watch script against a comparable cluster and characterize the behavior as the watch count approaches 9,000–10,000. Done means determining whether a hard limit exists and documenting the applicable configuration or limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, javascript, kubernetes
- Domain
- api, backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100