Highlight file descriptor limit as an operational concern
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
From https://forums.foundationdb.org/t/all-coordinators-crashed-at-same-time/1537 , FDB offers no warning about the number of open connections approaching the file descriptor limit, and potentially bad behavior (crashing, which essentially leads to crashlooping) when it does.
Going back through my notes from previous weeks, this apparently dropped off my mind, but is an important thing for us to address. I'd fully support cherrypicking the work for this back to 6.2 if it is done while we're still making 6.2 release. It's possible that the connection scaling work done for 6.2 would make this less of a potential problem?
I think there's multiple pieces of this that need to be addressed:
- [ ] Each process should publish as part of its status the number of connections vs file descriptor limit
- [ ] fdbcli status should include a warning if the file descriptor limit is the default 1024, or if the number of connections is approaching the limit
- [ ] Add to the documentation about setting up a cluster some text about raising file descriptor limits
- [ ] Improve the error message logged when fdbserver quits due to hitting the file descriptor limit
- [ ] Reconsider if fdbserver crashing if the file descriptor limit has been reached is the right thing to do
Contributor guide
Research direction
Start with the fdbcli status output and the fdbserver process status and shutdown behavior described in the checklist. Trace how connection counts and file descriptor limits are surfaced, then review the cluster setup documentation; done means the status warning, improved quit message, documentation guidance, and crash behavior are addressed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100