influxdata / influxdata/influxdb

InfluxDB holds TCP connections open to disconnected telegraf agents

Open
#14,830 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.

1. Run influxdb
2. Run telegraf from a number of remote hosts
3. Stop remote hosts, start, stop (repeat a few times)
4. Run `lsof -P | grep influx | awk -F':8086' '{print $2}' | awk -F':' '{print $1}' | sort | uniq -c | sort -nk 1` on the influx host to see increasing number of open connections to influx from telegraf agents

__Expected behavior:__
When telegraf shuts down, influx closes open connections

__Actual behavior:__
Influx continues to hold connections open until open file handle limits are reached

__Environment info:__

* System info: `Linux 4.14.109-80.92.amzn1.x86_64 x86_64`
* InfluxDB version: `InfluxDB v1.7.7 (git: 1.7 f8fdf652f348fc9980997fe1c972e2b79ddd13b0)`
* Other relevant environment details: Running on EC2 m5.xlarge

__Config:__
To my knowledge, we have no custom config settings. I am happy to provide any options if specific values are useful.

__Sample lsof output:__
```
18 ->widget-i-057bf12b491e0e34b.dev
19 ->build-i-08f663f229cb8c4a6.dev
19 ->cortex-i-03c69377b8cd3ff64.dev
19 ->frontdocker_manager-i-0c88604c9074d091b.dev
19 ->livescrape-i-0887216b1044ebae1.dev
19 ->mail2tix-i-0a4b0b7adee0b3781.dev
19 ->route-i-0f6cdabe9a9f9985a.dev
19 ->routev2-i-09e584b79f68c5047.dev
19 ->sapi-i-02c61a320842039aa.dev
19 ->session-i-072b25e4dd0e3a25d.dev
19 ->site-i-0bdce76332b7dfec9.dev
19 ->static-i-0f342c1dbf8671e0c.dev
19 ->storage-i-02979f25bbb505687.dev
19 ->txn-i-02ef77f68772dab44.dev
19 ->uiweb-i-01d51a1aff1a817dc.dev
19 ->wapi-i-03c932cbd6ae7fee0.dev
20 ->cdedocker-i-034b06ba85cbcd041.dev
20 ->frontdocker-i-06760dffd61212d0d.dev
20 ->middocker-i-0d3cf5df7aa445fad.dev
20 ->mongo-i-01024efdb7ae846ad.dev
20 ->mongo-i-069cff890881443d4.dev
20 ->seo-i-0df60a00156b5c4f0.dev
23 ->scheduler-i-09a8440089c187bb4.ci
24 ->scheduler-i-0007248c09b8dd029.dev
24 ->scheduler-i-0264a79cb9e6c261f.stage
24 ->scheduler-i-0a371f43677d77710.pilot
31 ->scheduler-i-0dfd32be39e68f4b7.dev
```

Our non-prod hosts shutdown at night, leaving connections open. Prod hosts do not shutdown and their connection counts stay at 1

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported behavior with InfluxDB and multiple remote telegraf agents, then inspect the server's TCP connection handling and existing connection-related tests or entry points. Done means connections from stopped agents are released instead of accumulating until the open-file limit is reached, while active agents retain their connections.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux
Domain
databases, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.