hashicorp / hashicorp/consul

Statsite telemetry leaves socket lingering to CLOSE_WAIT after remote server restarted

Open
#3,697 1 comment 0 reactions 0 assignees View on GitHub
theme/telemetry type/bug
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

### `consul version` for both Client and Server
Client: N/A
Server: Consul v0.9.3

### Operating system and Environment details
Running as Docker container based on Debian 8.9 image
Consul container is running in AWS on top of ECS (can't reproduce in local environment)
Consul connects to telemetry server via an ELB
```
...
"telemetry": {
"statsite_address": "xyz:8125"
},
...
```
### Description of the Issue (and unexpected/desired result)
When the telemetry server is restarted / killed / somehow temporarily unavailable Consul will not reconnect until the operating system takes care of the TCP socket left in CLOSE_WAIT state. By default this is in Linux I guess something over 2 hours:
```
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_time = 7200
```

Couldn't get any logs from this specific module (https://github.com/hashicorp/consul/blob/v0.9.3/vendor/github.com/armon/go-metrics/statsite.go), even though TRACE log levels enabled.

### Reproduction steps
1. Only reproducible in AWS, not locally with direct connection from a Docker container to another
2. Telemetry server is restarted / killed / somehow temporarily unavailable
3. Consul doesn't automatically reconnect, but leaves the connection lingering to a CLOSE_WAIT state:
```
tcp 1 0 10.x.y.z:44764 10.x.y.w:8125 CLOSE_WAIT 19920/consul
```
4. Restarting Consul will of course reconnect to telemetry server in :8125

### Log Fragments or Link to [gist](https://gist.github.com/)
Couldn't get any logs related to the statsite, even with TRACE level.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.