influxdata / influxdata/influxdb

Replication: Too many open files

Open
#23,996 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.

1. Create a Replication Stream
2. Disconnect the remote database so that data is written to replication queue
3. Write entries to the local database (Potentially greater than ulimit number of entries?)
4. Reconnect remote

__Expected behavior:__
Describe what you expected to happen.
No errors/Data should be replicated without issue.

__Actual behavior:__
Describe What actually happened.
Replication occurs until the influxdb process reaches ulimit.
Logs show it continues to retry. Replication hangs until TCP connections close.
Cycle repeats.

__Environment info:__
* System info: Run `uname -srm` and copy the output here
Linux 5.15.0-52-generic x86_64
* InfluxDB version: Run `influxd version` and copy the output here
InfluxDB v2.6.0 (git: 24a2b621ea) build_date: 2022-12-15T18:47:00Z
* Other relevant environment details: Container runtime, disk info, etc
```bash
$ ulimit -n
1024
```

__Logs:__
Include snippet of errors in log.

2022-12-20T03:06:51.311605Z error Error in replication stream {"log_id": "0esNoDGW000", "service": "replications", "replication_id": "0a77499ad3a3f000", "error": "Post \"http://localhost:8086/api/v2/write?bucket=77afb3a8286e1929&org=b7003017570b8fbd\": dial tcp localhost:8086: socket: too many open files", "retries": 1}

2022-12-20T03:06:51.728569Z info http: Accept error: accept tcp [::]:8086: accept4: too many open files; retrying in 5ms {"log_id": "0esNoDGW000", "service": "http"}

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the replication-stream sequence: disconnect the remote database, queue local writes, reconnect it, and observe the replication and HTTP logs under the stated ulimit. Trace the replication retry and connection handling involved in the reported socket exhaustion; done means queued data replicates after reconnection without exhausting file descriptors or hanging the server.

Written by the indexing model from the issue text.

Assessment

Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.