influxdata / influxdata/influxdb

subscriber: use http keep alive

Open
#15,932 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Proposal:__
InfluxDB should use permanent http connections to subscribed clients, to avoid network limitations.

__Current behavior:__
When Kapacitor is subscribed to influxdb, there's many show living http connections that transport only a few data points. At first, this looks like a waste of resources, but at high throughput rates, it has the side effect of running out of ports:
```dial tcp 127.0.0.1:9092: connect: cannot assign requested address```

This is because there's a max of 32k ports per 4mins (=2MSL) that can be assigned, see https://stackoverflow.com/questions/33177370/what-could-cause-so-many-time-wait-connections-to-be-open.

I've seen this with "stupid" clients that have no proper batching, but as they come with many different source adresses there's no problem on the external side, only when influxdb is talking to kapacitor, where all connections originate from the same ip address.

__Desired behavior:__
The subscriber code should keep the connection to it's subscribers alive.

__Alternatives considered:__
It could also be nice to batch the data points, even when the original clients do not batch them.

__Use case:__
This is an actual limit the the max write rate that can be processed.

Contributor guide

Open the contributing guide

Research direction

The issue points to the subscriber code but names no files or tests; start by locating that implementation and tracing how connections to subscribed clients are opened and closed. Done means subscriber connections remain alive and the reported port-exhaustion behavior is avoided, with coverage for the connection lifecycle.

Written by the indexing model from the issue text.

Assessment

Domain
backend, networking
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.