the response of the consul query service is slow, sometimes it takes several minutes to return
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
We have encountered a problem recently. The disk io on the cluster nodes is high for several minutes(cpu usage sometimes around 80%), and the response of the consul query service is slow, sometimes it takes several minutes to return. According to the description on the consul official website, Servers are generally I/O bound for writes and Servers are generally CPU bound for reads. Why does high disk io lead to slow query service response?
go-client send request to consul-server: ip:port/v1/health/service
disk_ioawait monitor:

For example, when it reaches 80ms, there will be a consul response exceeding 5s
consul-info:
```shell
"build": {
"prerelease": "",
"revision": "b3c2d385",
"version": "1.6.10"
},
"consul": {
"acl": "disabled",
"bootstrap": "false",
"known_datacenters": "1",
"leader": "false",
"leader_addr": "194.246.40.4:1114",
"server": "true"
},
```
Contributor guide
Research direction
Start with the Go client request to Consul's /v1/health/service endpoint and the reported disk_ioawait measurements. Compare the slow responses with the supplied Consul 1.6.10 server details and determine whether the disk I/O and query latency are causally related. Done means documenting a reproducible cause and the relevant remediation or next diagnostic step.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- api, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100