apple / apple/foundationdb

Highlight file descriptor limit as an operational concern

Open
#2,077 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

From https://forums.foundationdb.org/t/all-coordinators-crashed-at-same-time/1537 , FDB offers no warning about the number of open connections approaching the file descriptor limit, and potentially bad behavior (crashing, which essentially leads to crashlooping) when it does.

Going back through my notes from previous weeks, this apparently dropped off my mind, but is an important thing for us to address. I'd fully support cherrypicking the work for this back to 6.2 if it is done while we're still making 6.2 release. It's possible that the connection scaling work done for 6.2 would make this less of a potential problem?

I think there's multiple pieces of this that need to be addressed:

- [ ] Each process should publish as part of its status the number of connections vs file descriptor limit
- [ ] fdbcli status should include a warning if the file descriptor limit is the default 1024, or if the number of connections is approaching the limit
- [ ] Add to the documentation about setting up a cluster some text about raising file descriptor limits
- [ ] Improve the error message logged when fdbserver quits due to hitting the file descriptor limit
- [ ] Reconsider if fdbserver crashing if the file descriptor limit has been reached is the right thing to do

Contributor guide

Open the contributing guide

Research direction

Start with the fdbcli status output and the fdbserver process status and shutdown behavior described in the checklist. Trace how connection counts and file descriptor limits are surfaced, then review the cluster setup documentation; done means the status warning, improved quit message, documentation guidance, and crash behavior are addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.