canonical / canonical/microceph

Unable to scale from 3 to 2 cluster members without force

Open
#586 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
396
Forks
74
Avg merge
2d 20h
Merged PRs (30d)
7

Description

Affected version: `squid/stable`

A three member MicroCeph cluster (each node with one OSD) cannot be scaled down to two members because MicroCeph fails with the following error:

```bash
root@micro01:~# microceph cluster remove micro03
Error: Failed to execute pre-remove hook on cluster member "micro03": Need at least 3 mon, 1 mds, and 1 mgr besides micro03
```

Also see this conversation for reference https://chat.canonical.com/canonical/pl/i6pj4hyrejfeiyqqd1psrmo6we.

Reproducer steps:

* Start with a MicroCeph cluster having three members
* Add an OSD on each of the members
* ```bash
root@micro01:~# microceph cluster remove micro03 # Error is expected
Error: Failed to execute pre-remove hook on cluster member "micro03": Node micro03 still has disks configured, remove before proceeding
```
* ```bash
root@micro01:~# microceph disk list
Disks configured in MicroCeph:
+-----+----------+----------------------------------------------------+
| OSD | LOCATION | PATH |
+-----+----------+----------------------------------------------------+
| 1 | micro02 | /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_lxd_disk1 |
+-----+----------+----------------------------------------------------+
| 2 | micro03 | /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_lxd_disk1 |
+-----+----------+----------------------------------------------------+
| 3 | micro01 | /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_lxd_disk1 |
+-----+----------+----------------------------------------------------+
```
* ```bash
root@micro01:~# microceph disk remove 2 --confirm-failure-domain-downgrade # looks like --bypass-safety-checks is missing
Removing osd.2, timeout 1800s
Error: failed to remove disk: Failed to send request to target "micro03": cannot remove osd.2 we need at least 3 OSDs, have 3
```
* ```bash
root@micro01:~# microceph disk remove 2 --confirm-failure-domain-downgrade --bypass-safety-checks # Succeeds
Removing osd.2, timeout 1800s
```
* ```bash
root@micro01:~# microceph cluster remove micro03 # Still failing, no flags left besides --force
Error: Failed to execute pre-remove hook on cluster member "micro03": Need at least 3 mon, 1 mds, and 1 mgr besides micro03
```

Only when using `microceph cluster remove --force` the cluster member can be removed.

We (cc @mseralessandri) observed this issue whilst trying to remove the third node from a three node MicroCloud deployment which currently does not work as this seems to be blocked by MicroCeph.
The temporary workaround for now is to use the force flag with `microcloud remove --force`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.