apple / apple/foundationdb

Provide better insight into effectiveness of the fdbcli kill command

Open
#2,210 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

The `kill` command in `fdbcli` attempts to kill all the processes in a cluster with minimal guarantees about whether it will be able to do so. It also doesn't provide any information about how successful it was, which means that you either have to verify that processes were killed through some other channel or live with the risk that a kill might not have taken place on some processes.

In order to make it easier to determine the outcome of a kill, I think we should augment it to do something like the following:

1. Get an ID for each process that we intend to kill. This ID needs to stable for the lifetime of the process but must then change when the process restarts. This could, for example, be the ID of the worker interface.
2. Report the list of workers that we have IDs for to the caller (this is what we do already as output of the `kill` command). This allows us to verify that all workers are present if we use the `kill all` command.
3. Send the kill request to the desired processes.
4. Get the IDs of all workers present in the cluster. Each worker will be in one of three states -- ID is changed, ID is unchanged, or the worker is not present. The cluster may also have failed to recover, in which case we won't know the IDs of anybody.
5. Either retry or report an error for any worker that is no longer present or has an unchanged ID.

When dealing with cluster upgrades, this approach should work if we have multi-version CLI support (#1809). If we don't, then we could attempt to track a 4th state in step 4 above, which is that the worker is incompatible with the version of `fdbcli` that issued the kill. The caller would then interpret this to be acceptable if the protocol version of the cluster was expected to change due to upgrade.

In the case where the cluster is not recovering, we may want to allow some amount of time to recover before giving up.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the fdbcli kill command and how it discovers worker interfaces and sends kill requests. Review the multi-version CLI context in issue #1809. Done means the command can report worker outcomes and retry or report errors for missing or unchanged workers, including recovery delays.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
cli, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.