lablup / lablup/backend.ai

Multi-cluster Configuration Shows Incorrect Status When a Cluster Shuts Down

Open
#3,670 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

## Description

When operating a multi-cluster configuration in [http://Backend.AI](http://Backend.AI) , there seems to be an issue with how cluster statuses are reported. Specifically, in an 8-cluster setup, if one cluster shuts down, the overall status still displays as `running` instead of updating to `degraded`. This makes it difficult for users to identify issues since the actual state of the clusters does not reflect accurately. Users have to deduce on their own that a problem exists due to not all kernels being active as expected.

## Expected Behavior

The expected behavior in such scenarios is twofold:

1. The status should automatically update to `degraded` if any of the clusters within a multi-cluster setup shuts down or becomes unresponsive.
1. Implement a healing mechanism that either restarts all containers or only the ones that are missing, to ensure the integrity and the expected functionality of the multi-cluster setup.

## Steps to Reproduce

1. Set up a multi-cluster configuration.
1. Shut down or disconnect one of the clusters.
1. Observe that the overall status of the multi-cluster setup remains as `running`.

## Possible Solution

Implementing a monitoring and healing mechanism that can accurately detect the status of each cluster and take necessary actions such as updating the status to `degraded` and initiating a restart of the containers (either all or the ones that are missing) could be a potential solution.

This issue is critical for maintaining the reliability and usability of [http://Backend.AI](http://Backend.AI) in multi-cluster environments, and addressing it would greatly enhance user experience by providing a more accurate system state and automating recovery processes.

JIRA Issue: BA-723

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Reproduce the issue with an eight-cluster configuration by shutting down or disconnecting one cluster, then trace how the aggregate status is determined. Done means the status becomes degraded for a missing or unresponsive cluster and the intended container recovery behavior is defined and verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
backend, distributed-systems, observability
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.