canonical / canonical/microceph

It's not possible to rebuild a cluster after node failure

Open
#350 2 comments 6 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
396
Forks
74
Avg merge
2d 20h
Merged PRs (30d)
7

Description

# Issue report
Testing Microceph on a three node cluster. Removing a node (to simulate a failure) and rebuilding it, it's not possible to rejoin the cluster. There's no way to remove the OSDs from the failed node as this attempts to contact the node that failed (using microceph disk remove). Without being able to remove the OSDs, it's not possible to remove the failed node from the cluster (using microceph cluster remove).

## What version of MicroCeph are you using ?
18.2.0+snap71f71782c5

## What are the steps to reproduce this issue ?

1. Install Microceph on three nodes
3. Remove one of the nodes to simulate a node failing.
4. Unable to remove the failed node from Microceph since removing OSDs tries to contact the failed node.

## What happens (observed behaviour) ?
Unable to rejoin the node since Microceph thinks the node already exists.

## What were you expecting to happen ?

## Relevant logs, error output, etc.

If it’s considerably long, please paste to https://gist.github.com/ and insert the link here.

## Additional comments.

Contributor guide

Open the contributing guide

Research direction

Reproduce the three-node failure scenario described in the issue, then trace how `microceph disk remove` handles OSDs on an unavailable node and how `microceph cluster remove` handles that node. Check the relevant command entry points and existing tests for node removal. Done means a failed node can be removed and rebuilt so it can rejoin the cluster without contacting the unavailable node.

Written by the indexing model from the issue text.

Assessment

Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.