canonical / canonical/microcluster

More fine-grained statuses for MicroCluster

Open
#311 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
33
Forks
39
Avg merge
4h 49m
Merged PRs (30d)
6

Description

MicroCluster requires **all** nodes in a cluster to have the same schema/API version and if update will be triggered when one of the nodes is offline, the update will never finish.

I was thinking if we can prevent such update from happening by adding some guard rails (deb `preinst` or snap `pre-refresh`) before doing an update? But it seems to be tricky.

1. `member1` checks cluster status before upgrade, if **all** `online`, then it proceeds
```
+---------+-------------+
| NAME | STATUS |
+---------+-------------+
| member1 | ONLINE |
+---------+-------------+
| member2 | ONLINE |
+---------+-------------+
| member3 | ONLINE |
+---------+-------------+
```

2. `member1` is updated

cluster status from `member1`
```
+---------+---------------+
| NAME | STATUS |
+---------+---------------+
| member1 | UPGRADING |
+---------+---------------+
| member2 | NEEDS UPGRADE |
+---------+---------------+
| member3 | NEEDS UPGRADE |
+---------+---------------+
```

cluster status from `member2`
```
+---------+-------------+
| NAME | STATUS |
+---------+-------------+
| member1 | UNREACHABLE |
+---------+-------------+
| member2 | ONLINE |
+---------+-------------+
| member3 | ONLINE |
+---------+-------------+
```

Any reason the heartbeat endpoint couldn't be made to return a status indicating the `member1` is online but waiting for upgrade instead of unreachable?

P.S. I am using this as a current workaround:
```go
var healthCmd = rest.Endpoint{
Path: "health",
AllowedBeforeInit: true,
Post: rest.EndpointAction{Handler: cmdPost, AllowUntrusted: true},
}

func cmdPost(state state.State, r *http.Request) response.Response {
return response.SyncResponse(true, state.Database().Status())
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the heartbeat and health endpoints, including the cmdPost handler and state.Database().Status() shown in the workaround. Determine how a node that is online but waiting for upgrade is currently reported and define the additional status needed during a rolling upgrade. Done means cluster status reports that state consistently instead of marking the upgraded member unreachable.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend-api-design, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.