influxdata / influxdata/influxdb
Replication: Too many open files
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.
1. Create a Replication Stream
2. Disconnect the remote database so that data is written to replication queue
3. Write entries to the local database (Potentially greater than ulimit number of entries?)
4. Reconnect remote
__Expected behavior:__
Describe what you expected to happen.
No errors/Data should be replicated without issue.
__Actual behavior:__
Describe What actually happened.
Replication occurs until the influxdb process reaches ulimit.
Logs show it continues to retry. Replication hangs until TCP connections close.
Cycle repeats.
__Environment info:__
* System info: Run `uname -srm` and copy the output here
Linux 5.15.0-52-generic x86_64
* InfluxDB version: Run `influxd version` and copy the output here
InfluxDB v2.6.0 (git: 24a2b621ea) build_date: 2022-12-15T18:47:00Z
* Other relevant environment details: Container runtime, disk info, etc
```bash
$ ulimit -n
1024
```
__Logs:__
Include snippet of errors in log.
2022-12-20T03:06:51.311605Z error Error in replication stream {"log_id": "0esNoDGW000", "service": "replications", "replication_id": "0a77499ad3a3f000", "error": "Post \"http://localhost:8086/api/v2/write?bucket=77afb3a8286e1929&org=b7003017570b8fbd\": dial tcp localhost:8086: socket: too many open files", "retries": 1}
2022-12-20T03:06:51.728569Z info http: Accept error: accept tcp [::]:8086: accept4: too many open files; retrying in 5ms {"log_id": "0esNoDGW000", "service": "http"}
Contributor guide
Research direction
Start by reproducing the replication-stream sequence: disconnect the remote database, queue local writes, reconnect it, and observe the replication and HTTP logs under the stated ulimit. Trace the replication retry and connection handling involved in the reported socket exhaustion; done means queued data replicates after reconnection without exhausting file descriptors or hanging the server.
Written by the indexing model from the issue text.
Assessment
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100