canonical / canonical/microceph
Unable to scale from 3 to 2 cluster members without force
- Dominant language
- Go
- Stars
- 396
- Forks
- 74
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 7
Description
Affected version: `squid/stable`
A three member MicroCeph cluster (each node with one OSD) cannot be scaled down to two members because MicroCeph fails with the following error:
```bash
root@micro01:~# microceph cluster remove micro03
Error: Failed to execute pre-remove hook on cluster member "micro03": Need at least 3 mon, 1 mds, and 1 mgr besides micro03
```
Also see this conversation for reference https://chat.canonical.com/canonical/pl/i6pj4hyrejfeiyqqd1psrmo6we.
Reproducer steps:
* Start with a MicroCeph cluster having three members
* Add an OSD on each of the members
* ```bash
root@micro01:~# microceph cluster remove micro03 # Error is expected
Error: Failed to execute pre-remove hook on cluster member "micro03": Node micro03 still has disks configured, remove before proceeding
```
* ```bash
root@micro01:~# microceph disk list
Disks configured in MicroCeph:
+-----+----------+----------------------------------------------------+
| OSD | LOCATION | PATH |
+-----+----------+----------------------------------------------------+
| 1 | micro02 | /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_lxd_disk1 |
+-----+----------+----------------------------------------------------+
| 2 | micro03 | /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_lxd_disk1 |
+-----+----------+----------------------------------------------------+
| 3 | micro01 | /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_lxd_disk1 |
+-----+----------+----------------------------------------------------+
```
* ```bash
root@micro01:~# microceph disk remove 2 --confirm-failure-domain-downgrade # looks like --bypass-safety-checks is missing
Removing osd.2, timeout 1800s
Error: failed to remove disk: Failed to send request to target "micro03": cannot remove osd.2 we need at least 3 OSDs, have 3
```
* ```bash
root@micro01:~# microceph disk remove 2 --confirm-failure-domain-downgrade --bypass-safety-checks # Succeeds
Removing osd.2, timeout 1800s
```
* ```bash
root@micro01:~# microceph cluster remove micro03 # Still failing, no flags left besides --force
Error: Failed to execute pre-remove hook on cluster member "micro03": Need at least 3 mon, 1 mds, and 1 mgr besides micro03
```
Only when using `microceph cluster remove --force` the cluster member can be removed.
We (cc @mseralessandri) observed this issue whilst trying to remove the third node from a three node MicroCloud deployment which currently does not work as this seems to be blocked by MicroCeph.
The temporary workaround for now is to use the force flag with `microcloud remove --force`.
Contributor guide
Assessment
This issue has not been assessed yet.