Provide better insight into effectiveness of the fdbcli kill command
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
The `kill` command in `fdbcli` attempts to kill all the processes in a cluster with minimal guarantees about whether it will be able to do so. It also doesn't provide any information about how successful it was, which means that you either have to verify that processes were killed through some other channel or live with the risk that a kill might not have taken place on some processes.
In order to make it easier to determine the outcome of a kill, I think we should augment it to do something like the following:
1. Get an ID for each process that we intend to kill. This ID needs to stable for the lifetime of the process but must then change when the process restarts. This could, for example, be the ID of the worker interface.
2. Report the list of workers that we have IDs for to the caller (this is what we do already as output of the `kill` command). This allows us to verify that all workers are present if we use the `kill all` command.
3. Send the kill request to the desired processes.
4. Get the IDs of all workers present in the cluster. Each worker will be in one of three states -- ID is changed, ID is unchanged, or the worker is not present. The cluster may also have failed to recover, in which case we won't know the IDs of anybody.
5. Either retry or report an error for any worker that is no longer present or has an unchanged ID.
When dealing with cluster upgrades, this approach should work if we have multi-version CLI support (#1809). If we don't, then we could attempt to track a 4th state in step 4 above, which is that the worker is incompatible with the version of `fdbcli` that issued the kill. The caller would then interpret this to be acceptable if the protocol version of the cluster was expected to change due to upgrade.
In the case where the cluster is not recovering, we may want to allow some amount of time to recover before giving up.
Contributor guide
Research direction
Start by tracing the fdbcli kill command and how it discovers worker interfaces and sends kill requests. Review the multi-version CLI context in issue #1809. Done means the command can report worker outcomes and retry or report errors for missing or unchanged workers, including recovery delays.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- cli, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100