apple / apple/foundationdb

Add capability for cluster to report desired process replacements in status

Open
#4,784 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

We could imagine wanting to build some automation around the replacing of processes in a cluster if they have failed. One way to do this now is to look for processes that are missing from status that we expect to be there, and after some amount of time replace them. However, there are also cases where processes are currently part of a cluster but not participating because of some other issue.

For example, we may have processes that have had IO errors and are no longer running their associated roles. Or possibly we've detected that a transaction log, etc. has become degraded and we don't want to use it. I'm sure there are other types of problems we could think about adding detection schemes for as well.

To support automation (or even manual management), it would be helpful if a cluster could report out those processes which it deemed needing replacement.

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Start by reviewing the existing cluster status output and process or role health detection. Define how failed, non-participating, or degraded processes are identified and exposed, then add coverage showing that operators can find processes needing replacement.

Written by the indexing model from the issue text.

Assessment

Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.