Multi-cluster Configuration Shows Incorrect Status When a Cluster Shuts Down
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
## Description
When operating a multi-cluster configuration in [http://Backend.AI](http://Backend.AI) , there seems to be an issue with how cluster statuses are reported. Specifically, in an 8-cluster setup, if one cluster shuts down, the overall status still displays as `running` instead of updating to `degraded`. This makes it difficult for users to identify issues since the actual state of the clusters does not reflect accurately. Users have to deduce on their own that a problem exists due to not all kernels being active as expected.
## Expected Behavior
The expected behavior in such scenarios is twofold:
1. The status should automatically update to `degraded` if any of the clusters within a multi-cluster setup shuts down or becomes unresponsive.
1. Implement a healing mechanism that either restarts all containers or only the ones that are missing, to ensure the integrity and the expected functionality of the multi-cluster setup.
## Steps to Reproduce
1. Set up a multi-cluster configuration.
1. Shut down or disconnect one of the clusters.
1. Observe that the overall status of the multi-cluster setup remains as `running`.
## Possible Solution
Implementing a monitoring and healing mechanism that can accurately detect the status of each cluster and take necessary actions such as updating the status to `degraded` and initiating a restart of the containers (either all or the ones that are missing) could be a potential solution.
This issue is critical for maintaining the reliability and usability of [http://Backend.AI](http://Backend.AI) in multi-cluster environments, and addressing it would greatly enhance user experience by providing a more accurate system state and automating recovery processes.
JIRA Issue: BA-723
Contributor guide
Research direction
No files, tests, or entry points are named. Reproduce the issue with an eight-cluster configuration by shutting down or disconnecting one cluster, then trace how the aggregate status is determined. Done means the status becomes degraded for a missing or unresponsive cluster and the intended container recovery behavior is defined and verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- backend, distributed-systems, observability
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100