apple / apple/foundationdb

fdbcli kill failure with TLS possibly due to slow task

Open
#2,462 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

We've seen some instances of using `fdbcli` to kill processes in the cluster failing due to `fdbcli` being unable to establish connections to processes in the cluster. More specifically, it succeeded in killing only those processes that it was already connected to, which were the coordinators and the cluster controller.

There is another issue (#2209) that would have `fdbcli` connect to all processes prior to issuing the kill request, but unless we deal with the slow task it may run into similar troubles.

Specifically, the problem seems to be that when using TLS, the `ASIOReactor` spends a sizable amount of time in the react function per connection. When there are a lot of connections at once, this can take long enough that the processes we are connecting to time out the connection without having successfully established it completely.

Contributor guide

Open the contributing guide

Research direction

Start with the fdbcli process-kill flow and the ASIOReactor react function mentioned in the report. Investigate TLS connection establishment when many processes are contacted at once, and verify the kill operation can connect to and terminate all intended processes without connection timeouts.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
cli, distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.