When is safe (unsafe) to bounce a cluster
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
When a cluster has incidents, bouncing the cluster can usually(not always) resolve the problem.
If we know when it is safe (or unsafe) to bounce a cluster, we can let a FDB role (e.g., ratekeepr or a new role) automatically bounce the cluster. This can reduce the human intervention in operating a FDB cluster.
This involves two items:
- [ ] Identify when it is unsafe to bounce a cluster; (most challenging part I think.)
- [ ] Kick off the bounce (i.e., triggering recovery) when FDB incident happens.
Different orgs. may have slightly different metrics to define an incident. It is better for this feature to work on customized definition of incidents.
P.S.
As @ajbeamon points out, it is unsafe to bounce a cluster when the cluster is in the middle of recovery.
We may need a full list of this type of scenarios.
Contributor guide
Research direction
The issue names no files, tests, or concrete entry points. Start by defining the incident metrics and cataloging scenarios involving cluster recovery and bouncing, including the noted case where recovery is in progress. Done means the unsafe conditions are documented and a customized incident definition can trigger recovery without unsafe intervention.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100