Heavily loaded Consul agents might leak file descriptors
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
#### Overview of the Issue
We have regularly some heavily loaded Consul agents leaking file descriptors. This does not happen, but happens (~ once a week on 50000 agents).
In such case, ss command shows few connections, but lots of FDs are visible in `/proc/ socket:[426191617]
lrwx------ 1 consul consul 64 Nov 19 00:36 9641 -> socket:[426191620]
lrwx------ 1 consul consul 64 Nov 19 00:36 9642 -> socket:[426191623]
lrwx------ 1 consul consul 64 Nov 19 00:36 9643 -> socket:[426193111]
lrwx------ 1 consul consul 64 Nov 19 00:36 9644 -> socket:[426193115]
lrwx------ 1 consul consul 64 Nov 19 00:36 9645 -> socket:[426193121]
lrwx------ 1 consul consul 64 Nov 19 00:36 9646 -> socket:[426193125]
lrwx------ 1 consul consul 64 Nov 19 00:36 9647 -> socket:[426193127]
lrwx------ 1 consul consul 64 Nov 19 00:36 9648 -> socket:[426193131]
lrwx------ 1 consul consul 64 Nov 19 00:36 9649 -> socket:[426193137]
lrwx------ 1 consul consul 64 Nov 16 10:20 965 -> socket:[435058625]
lrwx------ 1 consul consul 64 Nov 19 00:36 9650 -> socket:[426193602]
lrwx------ 1 consul consul 64 Nov 19 00:36 9651 -> socket:[426193604]
lrwx------ 1 consul consul 64 Nov 19 00:36 9652 -> socket:[426193605]
lrwx------ 1 consul consul 64 Nov 19 00:36 9653 -> socket:[426193616]
lrwx------ 1 consul consul 64 Nov 19 00:36 9654 -> socket:[426193695]
lrwx------ 1 consul consul 64 Nov 19 00:36 9655 -> socket:[426194050]
lrwx------ 1 consul consul 64 Nov 19 00:36 9656 -> socket:[426194086]
lrwx------ 1 consul consul 64 Nov 19 00:36 9657 -> socket:[426194089]
lrwx------ 1 consul consul 64 Nov 19 00:36 9658 -> socket:[426194566]
lrwx------ 1 consul consul 64 Nov 19 00:36 9659 -> socket:[426194544]
lrwx------ 1 consul consul 64 Nov 16 10:20 966 -> socket:[435006470]
lrwx------ 1 consul consul 64 Nov 19 00:36 9660 -> socket:[426194575]
lrwx------ 1 consul consul 64 Nov 19 00:36 9661 -> socket:[426194585]
lrwx------ 1 consul consul 64 Nov 19 00:36 9662 -> socket:[426194593]
lrwx------ 1 consul consul 64 Nov 19 00:36 9663 -> socket:[426194597]
lrwx------ 1 consul consul 64 Nov 19 00:36 9664 -> socket:[426194603]
lrwx------ 1 consul consul 64 Nov 19 00:36 9665 -> socket:[426196097]
lrwx------ 1 consul consul 64 Nov 19 00:36 9666 -> socket:[426194964]
lrwx------ 1 consul consul 64 Nov 19 00:36 9667 -> socket:[426196105]
lrwx------ 1 consul consul 64 Nov 19 00:36 9668 -> socket:[426194967]
lrwx------ 1 consul consul 64 Nov 19 00:36 9669 -> socket:[426194969]
```
#### Reproduction Steps
Unknown, happens mostly on very loaded agents.
### Consul info for both Client and Server
Client info
```
agent:
check_monitors = 67
check_ttls = 0
checks = 166
services = 74
build:
prerelease = criteo1
revision =
version = 1.8.5
consul:
acl = legacy
known_servers = 10
server = false
runtime:
arch = amd64
cpu_count = 2
goroutines = 606
max_procs = 2
os = linux
version = go1.14.10
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 25299
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 2037594
members = 582
query_queue = 0
query_time = 2157
```
Server info
```
output from server 'consul info' command here
```
### Operating system and Environment details
CentOS 7, Consul Servers enterprise, Consul agent OSS 1.8.5
We also had this issue on previous versions 1.8.x
Happens on large DCs (more than 5k nodes) as well as smaller ones (500 nodes)
Contributor guide
Assessment
This issue has not been assessed yet.