apple / apple/foundationdb

When is safe (unsafe) to bounce a cluster

Open
#3,035 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

When a cluster has incidents, bouncing the cluster can usually(not always) resolve the problem.

If we know when it is safe (or unsafe) to bounce a cluster, we can let a FDB role (e.g., ratekeepr or a new role) automatically bounce the cluster. This can reduce the human intervention in operating a FDB cluster.

This involves two items:
- [ ] Identify when it is unsafe to bounce a cluster; (most challenging part I think.)
- [ ] Kick off the bounce (i.e., triggering recovery) when FDB incident happens.

Different orgs. may have slightly different metrics to define an incident. It is better for this feature to work on customized definition of incidents.

P.S.
As @ajbeamon points out, it is unsafe to bounce a cluster when the cluster is in the middle of recovery.
We may need a full list of this type of scenarios.

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or concrete entry points. Start by defining the incident metrics and cataloging scenarios involving cluster recovery and bouncing, including the noted case where recovery is in progress. Done means the unsafe conditions are documented and a customized incident definition can trigger recovery without unsafe intervention.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.