influxdata / influxdata/influxdb
subscriber: use http keep alive
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Proposal:__
InfluxDB should use permanent http connections to subscribed clients, to avoid network limitations.
__Current behavior:__
When Kapacitor is subscribed to influxdb, there's many show living http connections that transport only a few data points. At first, this looks like a waste of resources, but at high throughput rates, it has the side effect of running out of ports:
```dial tcp 127.0.0.1:9092: connect: cannot assign requested address```
This is because there's a max of 32k ports per 4mins (=2MSL) that can be assigned, see https://stackoverflow.com/questions/33177370/what-could-cause-so-many-time-wait-connections-to-be-open.
I've seen this with "stupid" clients that have no proper batching, but as they come with many different source adresses there's no problem on the external side, only when influxdb is talking to kapacitor, where all connections originate from the same ip address.
__Desired behavior:__
The subscriber code should keep the connection to it's subscribers alive.
__Alternatives considered:__
It could also be nice to batch the data points, even when the original clients do not batch them.
__Use case:__
This is an actual limit the the max write rate that can be processed.
Contributor guide
Research direction
The issue points to the subscriber code but names no files or tests; start by locating that implementation and tracing how connections to subscribed clients are opened and closed. Done means subscriber connections remain alive and the reported port-exhaustion behavior is avoided, with coverage for the connection lifecycle.
Written by the indexing model from the issue text.
Assessment
- Domain
- backend, networking
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100