hashicorp / hashicorp/consul

Consul consumes tens GB or RAM for no reason

Open
#16,290 4 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue

We use Consul in single-node mode for distributed locks and for service-discovery in our app. Service discovery used to connect our application environments with our monitoring. Consuls connect with each other over WAN.

After startup it starts consuming memory very quickly, memory usage goes over 30GB and quickly overwhelms our server. What I've found is that repeated calls to `consul info` shows that `goroutines` number increases rapidly along with increase of memory usage.

I'm also attaching logs.

I cap provide `consul debug` output by request of maintainers.

Possibly related issues: #12564, #9076, #12288, #3111
#### Reproduction Steps

So far I haven't figure out how to reproduce it in isolated environment.

### Consul info for both Client and Server

Server info

```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 1
build:
prerelease =
revision = 0e046bbb
version = 1.13.2
version_metadata =
consul:
acl = disabled
bootstrap = true
known_datacenters = 4
leader = true
leader_addr = 172.22.0.2:8300
server = true
raft:
applied_index = 14
commit_index = 14
fsm_pending = 0
last_contact = 0
last_log_index = 14
last_log_term = 2
last_snapshot_index = 0
last_snapshot_term = 0
latest_configuration = [{Suffrage:Voter ID:6a5bc7e2-7c9b-bcdd-23a8-a4f0d35dc3d2 Address:172.22.0.2:8300}]
latest_configuration_index = 0
num_peers = 0
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 2
runtime:
arch = amd64
cpu_count = 8
goroutines = 642723
max_procs = 8
os = linux
version = go1.18.1
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 1
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 1
health_score = 4
intent_queue = 0
left = 0
member_time = 182
members = 5
query_queue = 0
query_time = 1
/ # consul ^C
/ # consul info
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 1
build:
prerelease =
revision = 0e046bbb
version = 1.13.2
version_metadata =
consul:
acl = disabled
bootstrap = true
known_datacenters = 4
leader = true
leader_addr = 172.22.0.2:8300
server = true
raft:
applied_index = 15
commit_index = 15
fsm_pending = 0
last_contact = 0
last_log_index = 15
last_log_term = 2
last_snapshot_index = 0
last_snapshot_term = 0
latest_configuration = [{Suffrage:Voter ID:6a5bc7e2-7c9b-bcdd-23a8-a4f0d35dc3d2 Address:172.22.0.2:8300}]
latest_configuration_index = 0
num_peers = 0
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 2
runtime:
arch = amd64
cpu_count = 8
goroutines = 818620
max_procs = 8
os = linux
version = go1.18.1
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 1
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 1
health_score = 5
intent_queue = 0
left = 0
member_time = 182
members = 5
query_queue = 0
query_time = 1
/ # consul info
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 1
build:
prerelease =
revision = 0e046bbb
version = 1.13.2
version_metadata =
consul:
acl = disabled
bootstrap = true
known_datacenters = 4
leader = true
leader_addr = 172.22.0.2:8300
server = true
raft:
applied_index = 16
commit_index = 16
fsm_pending = 0
last_contact = 0
last_log_index = 16
last_log_term = 2
last_snapshot_index = 0
last_snapshot_term = 0
latest_configuration = [{Suffrage:Voter ID:6a5bc7e2-7c9b-bcdd-23a8-a4f0d35dc3d2 Address:172.22.0.2:8300}]
latest_configuration_index = 0
num_peers = 0
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 2
runtime:
arch = amd64
cpu_count = 8
goroutines = 1147773
max_procs = 8
os = linux
version = go1.18.1
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 1
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 2
health_score = 4
intent_queue = 0
left = 0
member_time = 182
members = 5
query_queue = 0
query_time = 1
```

### Operating system and Environment details

Ubuntu 20.04

### Log Fragments
https://gist.github.com/AndreiPashkin/0a95cdcb5e349c881ff4ee94af5f7b15

### Version
```
# consul version
Consul v1.13.2
Revision 0e046bbb
Build Date 2022-09-20T20:30:07Z
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)
```

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the linked log fragments and comparing the repeated `consul info` output, focusing on the rapidly increasing goroutine count and memory use. Request or run `consul debug` as mentioned, then use the collected evidence to identify the source of the growth and verify that Consul v1.13.2 no longer overwhelms the server.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.