hashicorp / hashicorp/consul

Heavily loaded Consul agents might leak file descriptors

Open
#9,233 3 comments 0 reactions 1 assignee Claimed by @zalimeni View on GitHub
type/bug
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

#### Overview of the Issue

We have regularly some heavily loaded Consul agents leaking file descriptors. This does not happen, but happens (~ once a week on 50000 agents).

In such case, ss command shows few connections, but lots of FDs are visible in `/proc/ socket:[426191617]
lrwx------ 1 consul consul 64 Nov 19 00:36 9641 -> socket:[426191620]
lrwx------ 1 consul consul 64 Nov 19 00:36 9642 -> socket:[426191623]
lrwx------ 1 consul consul 64 Nov 19 00:36 9643 -> socket:[426193111]
lrwx------ 1 consul consul 64 Nov 19 00:36 9644 -> socket:[426193115]
lrwx------ 1 consul consul 64 Nov 19 00:36 9645 -> socket:[426193121]
lrwx------ 1 consul consul 64 Nov 19 00:36 9646 -> socket:[426193125]
lrwx------ 1 consul consul 64 Nov 19 00:36 9647 -> socket:[426193127]
lrwx------ 1 consul consul 64 Nov 19 00:36 9648 -> socket:[426193131]
lrwx------ 1 consul consul 64 Nov 19 00:36 9649 -> socket:[426193137]
lrwx------ 1 consul consul 64 Nov 16 10:20 965 -> socket:[435058625]
lrwx------ 1 consul consul 64 Nov 19 00:36 9650 -> socket:[426193602]
lrwx------ 1 consul consul 64 Nov 19 00:36 9651 -> socket:[426193604]
lrwx------ 1 consul consul 64 Nov 19 00:36 9652 -> socket:[426193605]
lrwx------ 1 consul consul 64 Nov 19 00:36 9653 -> socket:[426193616]
lrwx------ 1 consul consul 64 Nov 19 00:36 9654 -> socket:[426193695]
lrwx------ 1 consul consul 64 Nov 19 00:36 9655 -> socket:[426194050]
lrwx------ 1 consul consul 64 Nov 19 00:36 9656 -> socket:[426194086]
lrwx------ 1 consul consul 64 Nov 19 00:36 9657 -> socket:[426194089]
lrwx------ 1 consul consul 64 Nov 19 00:36 9658 -> socket:[426194566]
lrwx------ 1 consul consul 64 Nov 19 00:36 9659 -> socket:[426194544]
lrwx------ 1 consul consul 64 Nov 16 10:20 966 -> socket:[435006470]
lrwx------ 1 consul consul 64 Nov 19 00:36 9660 -> socket:[426194575]
lrwx------ 1 consul consul 64 Nov 19 00:36 9661 -> socket:[426194585]
lrwx------ 1 consul consul 64 Nov 19 00:36 9662 -> socket:[426194593]
lrwx------ 1 consul consul 64 Nov 19 00:36 9663 -> socket:[426194597]
lrwx------ 1 consul consul 64 Nov 19 00:36 9664 -> socket:[426194603]
lrwx------ 1 consul consul 64 Nov 19 00:36 9665 -> socket:[426196097]
lrwx------ 1 consul consul 64 Nov 19 00:36 9666 -> socket:[426194964]
lrwx------ 1 consul consul 64 Nov 19 00:36 9667 -> socket:[426196105]
lrwx------ 1 consul consul 64 Nov 19 00:36 9668 -> socket:[426194967]
lrwx------ 1 consul consul 64 Nov 19 00:36 9669 -> socket:[426194969]
```

#### Reproduction Steps

Unknown, happens mostly on very loaded agents.

### Consul info for both Client and Server

Client info

```
agent:
check_monitors = 67
check_ttls = 0
checks = 166
services = 74
build:
prerelease = criteo1
revision =
version = 1.8.5
consul:
acl = legacy
known_servers = 10
server = false
runtime:
arch = amd64
cpu_count = 2
goroutines = 606
max_procs = 2
os = linux
version = go1.14.10
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 25299
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 2037594
members = 582
query_queue = 0
query_time = 2157
```

Server info

```
output from server 'consul info' command here
```

### Operating system and Environment details

CentOS 7, Consul Servers enterprise, Consul agent OSS 1.8.5

We also had this issue on previous versions 1.8.x

Happens on large DCs (more than 5k nodes) as well as smaller ones (500 nodes)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.