Consul consumes tens GB or RAM for no reason
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Overview of the Issue
We use Consul in single-node mode for distributed locks and for service-discovery in our app. Service discovery used to connect our application environments with our monitoring. Consuls connect with each other over WAN.
After startup it starts consuming memory very quickly, memory usage goes over 30GB and quickly overwhelms our server. What I've found is that repeated calls to `consul info` shows that `goroutines` number increases rapidly along with increase of memory usage.
I'm also attaching logs.
I cap provide `consul debug` output by request of maintainers.
Possibly related issues: #12564, #9076, #12288, #3111
#### Reproduction Steps
So far I haven't figure out how to reproduce it in isolated environment.
### Consul info for both Client and Server
Server info
```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 1
build:
prerelease =
revision = 0e046bbb
version = 1.13.2
version_metadata =
consul:
acl = disabled
bootstrap = true
known_datacenters = 4
leader = true
leader_addr = 172.22.0.2:8300
server = true
raft:
applied_index = 14
commit_index = 14
fsm_pending = 0
last_contact = 0
last_log_index = 14
last_log_term = 2
last_snapshot_index = 0
last_snapshot_term = 0
latest_configuration = [{Suffrage:Voter ID:6a5bc7e2-7c9b-bcdd-23a8-a4f0d35dc3d2 Address:172.22.0.2:8300}]
latest_configuration_index = 0
num_peers = 0
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 2
runtime:
arch = amd64
cpu_count = 8
goroutines = 642723
max_procs = 8
os = linux
version = go1.18.1
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 1
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 1
health_score = 4
intent_queue = 0
left = 0
member_time = 182
members = 5
query_queue = 0
query_time = 1
/ # consul ^C
/ # consul info
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 1
build:
prerelease =
revision = 0e046bbb
version = 1.13.2
version_metadata =
consul:
acl = disabled
bootstrap = true
known_datacenters = 4
leader = true
leader_addr = 172.22.0.2:8300
server = true
raft:
applied_index = 15
commit_index = 15
fsm_pending = 0
last_contact = 0
last_log_index = 15
last_log_term = 2
last_snapshot_index = 0
last_snapshot_term = 0
latest_configuration = [{Suffrage:Voter ID:6a5bc7e2-7c9b-bcdd-23a8-a4f0d35dc3d2 Address:172.22.0.2:8300}]
latest_configuration_index = 0
num_peers = 0
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 2
runtime:
arch = amd64
cpu_count = 8
goroutines = 818620
max_procs = 8
os = linux
version = go1.18.1
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 1
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 1
health_score = 5
intent_queue = 0
left = 0
member_time = 182
members = 5
query_queue = 0
query_time = 1
/ # consul info
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 1
build:
prerelease =
revision = 0e046bbb
version = 1.13.2
version_metadata =
consul:
acl = disabled
bootstrap = true
known_datacenters = 4
leader = true
leader_addr = 172.22.0.2:8300
server = true
raft:
applied_index = 16
commit_index = 16
fsm_pending = 0
last_contact = 0
last_log_index = 16
last_log_term = 2
last_snapshot_index = 0
last_snapshot_term = 0
latest_configuration = [{Suffrage:Voter ID:6a5bc7e2-7c9b-bcdd-23a8-a4f0d35dc3d2 Address:172.22.0.2:8300}]
latest_configuration_index = 0
num_peers = 0
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 2
runtime:
arch = amd64
cpu_count = 8
goroutines = 1147773
max_procs = 8
os = linux
version = go1.18.1
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 1
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 2
health_score = 4
intent_queue = 0
left = 0
member_time = 182
members = 5
query_queue = 0
query_time = 1
```
### Operating system and Environment details
Ubuntu 20.04
### Log Fragments
https://gist.github.com/AndreiPashkin/0a95cdcb5e349c881ff4ee94af5f7b15
### Version
```
# consul version
Consul v1.13.2
Revision 0e046bbb
Build Date 2022-09-20T20:30:07Z
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)
```
Contributor guide
Research direction
Start by reviewing the linked log fragments and comparing the repeated `consul info` output, focusing on the rapidly increasing goroutine count and memory use. Request or run `consul debug` as mentioned, then use the collected evidence to identify the source of the growth and verify that Consul v1.13.2 no longer overwhelms the server.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100