influxdata / influxdata/influxdb
InfluxDB holds TCP connections open to disconnected telegraf agents
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.
1. Run influxdb
2. Run telegraf from a number of remote hosts
3. Stop remote hosts, start, stop (repeat a few times)
4. Run `lsof -P | grep influx | awk -F':8086' '{print $2}' | awk -F':' '{print $1}' | sort | uniq -c | sort -nk 1` on the influx host to see increasing number of open connections to influx from telegraf agents
__Expected behavior:__
When telegraf shuts down, influx closes open connections
__Actual behavior:__
Influx continues to hold connections open until open file handle limits are reached
__Environment info:__
* System info: `Linux 4.14.109-80.92.amzn1.x86_64 x86_64`
* InfluxDB version: `InfluxDB v1.7.7 (git: 1.7 f8fdf652f348fc9980997fe1c972e2b79ddd13b0)`
* Other relevant environment details: Running on EC2 m5.xlarge
__Config:__
To my knowledge, we have no custom config settings. I am happy to provide any options if specific values are useful.
__Sample lsof output:__
```
18 ->widget-i-057bf12b491e0e34b.dev
19 ->build-i-08f663f229cb8c4a6.dev
19 ->cortex-i-03c69377b8cd3ff64.dev
19 ->frontdocker_manager-i-0c88604c9074d091b.dev
19 ->livescrape-i-0887216b1044ebae1.dev
19 ->mail2tix-i-0a4b0b7adee0b3781.dev
19 ->route-i-0f6cdabe9a9f9985a.dev
19 ->routev2-i-09e584b79f68c5047.dev
19 ->sapi-i-02c61a320842039aa.dev
19 ->session-i-072b25e4dd0e3a25d.dev
19 ->site-i-0bdce76332b7dfec9.dev
19 ->static-i-0f342c1dbf8671e0c.dev
19 ->storage-i-02979f25bbb505687.dev
19 ->txn-i-02ef77f68772dab44.dev
19 ->uiweb-i-01d51a1aff1a817dc.dev
19 ->wapi-i-03c932cbd6ae7fee0.dev
20 ->cdedocker-i-034b06ba85cbcd041.dev
20 ->frontdocker-i-06760dffd61212d0d.dev
20 ->middocker-i-0d3cf5df7aa445fad.dev
20 ->mongo-i-01024efdb7ae846ad.dev
20 ->mongo-i-069cff890881443d4.dev
20 ->seo-i-0df60a00156b5c4f0.dev
23 ->scheduler-i-09a8440089c187bb4.ci
24 ->scheduler-i-0007248c09b8dd029.dev
24 ->scheduler-i-0264a79cb9e6c261f.stage
24 ->scheduler-i-0a371f43677d77710.pilot
31 ->scheduler-i-0dfd32be39e68f4b7.dev
```
Our non-prod hosts shutdown at night, leaving connections open. Prod hosts do not shutdown and their connection counts stay at 1
Contributor guide
Research direction
Start by reproducing the reported behavior with InfluxDB and multiple remote telegraf agents, then inspect the server's TCP connection handling and existing connection-related tests or entry points. Done means connections from stopped agents are released instead of accumulating until the open-file limit is reached, while active agents retain their connections.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux
- Domain
- databases, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100