hashicorp / hashicorp/consul

the response of the consul query service is slow, sometimes it takes several minutes to return

Open
#17,140 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

We have encountered a problem recently. The disk io on the cluster nodes is high for several minutes(cpu usage sometimes around 80%), and the response of the consul query service is slow, sometimes it takes several minutes to return. According to the description on the consul official website, Servers are generally I/O bound for writes and Servers are generally CPU bound for reads. Why does high disk io lead to slow query service response?

go-client send request to consul-server: ip:port/v1/health/service
disk_ioawait monitor:
![image](https://user-images.githubusercontent.com/85682859/234459547-b422113c-697d-44da-b1ff-87d29c0f5ecb.png)
For example, when it reaches 80ms, there will be a consul response exceeding 5s

consul-info:
```shell
"build": {
"prerelease": "",
"revision": "b3c2d385",
"version": "1.6.10"
},
"consul": {
"acl": "disabled",
"bootstrap": "false",
"known_datacenters": "1",
"leader": "false",
"leader_addr": "194.246.40.4:1114",
"server": "true"
},
```

Contributor guide

Open the contributing guide

Research direction

Start with the Go client request to Consul's /v1/health/service endpoint and the reported disk_ioawait measurements. Compare the slow responses with the supplied Consul 1.6.10 server details and determine whether the disk I/O and query latency are causally related. Done means documenting a reproducible cause and the relevant remediation or next diagnostic step.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.