Consul agent cpu utilization is at 1575% and completely unresponsive
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Overview of the Issue
We have a consul running in a pod in azure kubernetes cluster, after several hours of running, the cpu usage of consul agent process in the container goes to 1575% and completely unresponsive. Any further request to consul is just hanging there. It's a single node consul. We have to restart the pod to resolve it. But the issue will happen again after some time (sometimes within half an hour, sometimes after several hours.)
#### Reproduction Steps
It only happens in the production, we can't reproduce it in other places.
### Operating system and Environment details
One weird thing we notice is the error `cache '/etc/ld.so.cache' is corrupt` when we run `consul -v`, not sure if this is the root cause. It's a read only file, so we can't remove it and rebuild it.
```
: cache '/etc/ld.so.cache' is corrupt
Consul v1.12.0
Revision 09a8cdb4
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically useprotocol >2 when speaking to compatible agents)
```
OS version:
```
[consul@consul-74d968487f-bdl6k /]$ cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.6 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.6"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
```
### Log Fragments
It doesn't log any more after it's frozen. I attached the files generated by `consul debug` which includes 2 minutes logs. These are generated around 10 minutes before the issue happens.
[consul-debug-2022-08-18T13-31-44Z.tar.gz](https://github.com/hashicorp/consul/files/9410481/consul-debug-2022-08-18T13-31-44Z.tar.gz)
Contributor guide
Research direction
Start by examining the attached consul-debug-2022-08-18T13-31-44Z.tar.gz and compare its two-minute logs with the reported Consul v1.12.0, RHEL 8.6, and Kubernetes environment. Investigate the CPU spike and unresponsive agent, including the reported /etc/ld.so.cache error, since the issue is not reproducible outside production. Done requires a confirmed cause and a validated fix or mitigation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, go, kubernetes, linux
- Domain
- cloud, devops, distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100