hashicorp / hashicorp/consul

Consul agent cpu utilization is at 1575% and completely unresponsive

Open
#14,319 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue

We have a consul running in a pod in azure kubernetes cluster, after several hours of running, the cpu usage of consul agent process in the container goes to 1575% and completely unresponsive. Any further request to consul is just hanging there. It's a single node consul. We have to restart the pod to resolve it. But the issue will happen again after some time (sometimes within half an hour, sometimes after several hours.)

#### Reproduction Steps

It only happens in the production, we can't reproduce it in other places.

### Operating system and Environment details

One weird thing we notice is the error `cache '/etc/ld.so.cache' is corrupt` when we run `consul -v`, not sure if this is the root cause. It's a read only file, so we can't remove it and rebuild it.

```
: cache '/etc/ld.so.cache' is corrupt
Consul v1.12.0
Revision 09a8cdb4
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically useprotocol >2 when speaking to compatible agents)
```

OS version:
```
[consul@consul-74d968487f-bdl6k /]$ cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.6 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.6"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
```

### Log Fragments

It doesn't log any more after it's frozen. I attached the files generated by `consul debug` which includes 2 minutes logs. These are generated around 10 minutes before the issue happens.
[consul-debug-2022-08-18T13-31-44Z.tar.gz](https://github.com/hashicorp/consul/files/9410481/consul-debug-2022-08-18T13-31-44Z.tar.gz)

Contributor guide

Open the contributing guide

Research direction

Start by examining the attached consul-debug-2022-08-18T13-31-44Z.tar.gz and compare its two-minute logs with the reported Consul v1.12.0, RHEL 8.6, and Kubernetes environment. Investigate the CPU spike and unresponsive agent, including the reported /etc/ld.so.cache error, since the issue is not reproducible outside production. Done requires a confirmed cause and a validated fix or mitigation.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, go, kubernetes, linux
Domain
cloud, devops, distributed-systems, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.