canonical / canonical/microcluster
More fine-grained statuses for MicroCluster
- Dominant language
- Go
- Stars
- 33
- Forks
- 39
- Avg merge
- 4h 49m
- Merged PRs (30d)
- 6
Description
MicroCluster requires **all** nodes in a cluster to have the same schema/API version and if update will be triggered when one of the nodes is offline, the update will never finish.
I was thinking if we can prevent such update from happening by adding some guard rails (deb `preinst` or snap `pre-refresh`) before doing an update? But it seems to be tricky.
1. `member1` checks cluster status before upgrade, if **all** `online`, then it proceeds
```
+---------+-------------+
| NAME | STATUS |
+---------+-------------+
| member1 | ONLINE |
+---------+-------------+
| member2 | ONLINE |
+---------+-------------+
| member3 | ONLINE |
+---------+-------------+
```
2. `member1` is updated
cluster status from `member1`
```
+---------+---------------+
| NAME | STATUS |
+---------+---------------+
| member1 | UPGRADING |
+---------+---------------+
| member2 | NEEDS UPGRADE |
+---------+---------------+
| member3 | NEEDS UPGRADE |
+---------+---------------+
```
cluster status from `member2`
```
+---------+-------------+
| NAME | STATUS |
+---------+-------------+
| member1 | UNREACHABLE |
+---------+-------------+
| member2 | ONLINE |
+---------+-------------+
| member3 | ONLINE |
+---------+-------------+
```
Any reason the heartbeat endpoint couldn't be made to return a status indicating the `member1` is online but waiting for upgrade instead of unreachable?
P.S. I am using this as a current workaround:
```go
var healthCmd = rest.Endpoint{
Path: "health",
AllowedBeforeInit: true,
Post: rest.EndpointAction{Handler: cmdPost, AllowUntrusted: true},
}
func cmdPost(state state.State, r *http.Request) response.Response {
return response.SyncResponse(true, state.Database().Status())
}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the heartbeat and health endpoints, including the cmdPost handler and state.Database().Status() shown in the workaround. Determine how a node that is online but waiting for upgrade is currently reported and define the additional status needed during a rolling upgrade. Done means cluster status reports that state consistently instead of marking the upgraded member unreachable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend-api-design, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100