canonical / canonical/microceph
It's not possible to rebuild a cluster after node failure
- Dominant language
- Go
- Stars
- 396
- Forks
- 74
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 7
Description
# Issue report
Testing Microceph on a three node cluster. Removing a node (to simulate a failure) and rebuilding it, it's not possible to rejoin the cluster. There's no way to remove the OSDs from the failed node as this attempts to contact the node that failed (using microceph disk remove). Without being able to remove the OSDs, it's not possible to remove the failed node from the cluster (using microceph cluster remove).
## What version of MicroCeph are you using ?
18.2.0+snap71f71782c5
## What are the steps to reproduce this issue ?
1. Install Microceph on three nodes
3. Remove one of the nodes to simulate a node failing.
4. Unable to remove the failed node from Microceph since removing OSDs tries to contact the failed node.
## What happens (observed behaviour) ?
Unable to rejoin the node since Microceph thinks the node already exists.
…
## What were you expecting to happen ?
…
## Relevant logs, error output, etc.
If it’s considerably long, please paste to https://gist.github.com/ and insert the link here.
## Additional comments.
…
Contributor guide
Research direction
Reproduce the three-node failure scenario described in the issue, then trace how `microceph disk remove` handles OSDs on an unavailable node and how `microceph cluster remove` handles that node. Check the relevant command entry points and existing tests for node removal. Done means a failed node can be removed and rebuilt so it can rejoin the cluster without contacting the unavailable node.
Written by the indexing model from the issue text.
Assessment
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100