canonical / canonical/microceph

microceph disk list does not list multipath disks

Open
#647 11 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
396
Forks
74
Avg merge
2d 20h
Merged PRs (30d)
7

Description

# Context
On a PoC environment we are using LUNs as disks for Ceph

This disk has two paths to reach it, sdb and sdc
Now, Microceph is only listing the first path sdb, instead of the the multipath device

The problem is that in multiplath you cannot mount a filesystem on sdb or sdc, it has to be on mpathX device. So installing any filesystem on top of sdb will fail

# Version
squid/stable: 19.2.1+snap74c0060321 2025-10-23 (1582) 128MB

# Evidence

## From MicroCeph
```bash
# ubuntu@msc-ostk-np-comp02:~$ sudo microceph disk list

Available unpartitioned disks on this system:
+--------+----------+------+----------------------------------------------------------------+
| MODEL | CAPACITY | TYPE | PATH |
+--------+----------+------+----------------------------------------------------------------+
| OPEN-V | 5.00TiB | scsi | /dev/disk/by-id/scsi-SHITACHI_OPEN-V_411515 |
+--------+----------+------+----------------------------------------------------------------+
| OPEN-V | 5.00TiB | scsi | /dev/disk/by-path/pci-0000:b1:00.0-fc-0x50060e8008a0bf21-lun-1 |
+--------+----------+------+----------------------------------------------------------------+
```

As expected, trying to add any of these disks will fail
```bash
# ubuntu@msc-ostk-np-comp02:~$ sudo microceph disk add /dev/disk/by-id/scsi-SHITACHI_OPEN-V_411515

+---------------------------------------------+---------+
| PATH | STATUS |
+---------------------------------------------+---------+
| /dev/disk/by-id/scsi-SHITACHI_OPEN-V_411515 | Failure |
+---------------------------------------------+---------+
Error: failed to bootstrap OSD: Failed to run: ceph-osd --mkfs --no-mon-config -i 1: exit status 250 (2025-11-05T21:30:45.659+0000 788fe0dd5640 -1 bdev(0x5a46261a4000 /var/lib/ceph/osd/ceph-1/block) open open got: (16) Device or resource busy
2025-11-05T21:30:45.659+0000 788fe0dd5640 -1 bdev(0x5a46261a4000 /var/lib/ceph/osd/ceph-1/block) open open got: (16) Device or resource busy
2025-11-05T21:30:45.659+0000 788fe0dd5640 -1 bluestore(/var/lib/ceph/osd/ceph-1) mkfs failed, (16) Device or resource busy
2025-11-05T21:30:45.659+0000 788fe0dd5640 -1 OSD::mkfs: ObjectStore::mkfs failed with error (16) Device or resource busy
2025-11-05T21:30:45.659+0000 788fe0dd5640 -1 ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-1: (16) Device or resource busy)
```

## From Linux point of view
```bash
# ubuntu@msc-ostk-np-comp02:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 50.9M 1 loop /snap/snapd/25577
loop1 7:1 0 63.8M 1 loop /snap/core20/2669
loop2 7:2 0 176.2M 1 loop /snap/k8s/4306
loop3 7:3 0 66.8M 1 loop /snap/core24/1225
loop4 7:4 0 122.2M 1 loop /snap/microceph/1601
sda 8:0 0 446.6G 0 disk
├─sda1 8:1 0 976M 0 part /boot/efi
└─sda2 8:2 0 445.7G 0 part /
sdb 8:16 0 5T 0 disk
└─mpatha 252:0 0 5T 0 mpath
sdc 8:32 0 5T 0 disk
└─mpatha 252:0 0 5T 0 mpath

# ubuntu@msc-ostk-np-comp02:~$ ls -l /dev/disk/by-id
total 0
lrwxrwxrwx 1 root root 10 Nov 4 21:09 dm-name-mpatha -> ../../dm-0
lrwxrwxrwx 1 root root 10 Nov 4 21:09 dm-uuid-mpath-360060e8008a0bf000050a0bf00000110 -> ../../dm-0
lrwxrwxrwx 1 root root 9 Nov 4 21:27 scsi-1HITACHI_5050A0BF0110 -> ../../sdb
lrwxrwxrwx 1 root root 10 Nov 4 21:09 scsi-360060e8008a0bf000050a0bf00000110 -> ../../dm-0
lrwxrwxrwx 1 root root 9 Nov 4 21:08 scsi-36f4ee08067176b0030816726d9e4f749 -> ../../sda
lrwxrwxrwx 1 root root 10 Nov 4 21:08 scsi-36f4ee08067176b0030816726d9e4f749-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Nov 4 21:08 scsi-36f4ee08067176b0030816726d9e4f749-part2 -> ../../sda2
lrwxrwxrwx 1 root root 9 Nov 4 21:08 scsi-SDELL_PERC_H755_Front_0049f7e4d926678130006b176780e04e -> ../../sda
lrwxrwxrwx 1 root root 10 Nov 4 21:08 scsi-SDELL_PERC_H755_Front_0049f7e4d926678130006b176780e04e-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Nov 4 21:08 scsi-SDELL_PERC_H755_Front_0049f7e4d926678130006b176780e04e-part2 -> ../../sda2
lrwxrwxrwx 1 root root 9 Nov 4 21:27 scsi-SHITACHI_OPEN-V_411515 -> ../../sdb
lrwxrwxrwx 1 root root 10 Nov 4 21:09 wwn-0x60060e8008a0bf000050a0bf00000110 -> ../../dm-0
lrwxrwxrwx 1 root root 9 Nov 4 21:08 wwn-0x6f4ee08067176b0030816726d9e4f749 -> ../../sda
lrwxrwxrwx 1 root root 10 Nov 4 21:08 wwn-0x6f4ee08067176b0030816726d9e4f749-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Nov 4 21:08 wwn-0x6f4ee08067176b0030816726d9e4f749-part2 -> ../../sda2

# root@msc-ostk-np-comp02:/home/ubuntu# ls -lha /dev/disk/by-dname/
total 0
drwxr-xr-x 2 root root 120 Nov 4 21:09 .
drwxr-xr-x 11 root root 220 Nov 4 21:18 ..
lrwxrwxrwx 1 root root 10 Nov 4 21:09 ceph-disk-osd01 -> ../../dm-0
lrwxrwxrwx 1 root root 9 Nov 4 21:08 sda -> ../../sda
lrwxrwxrwx 1 root root 10 Nov 4 21:08 sda-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Nov 4 21:08 sda-part2 -> ../../sda2
```

# Additional information
Equivalent to this bug in MAAS -> https://bugs.launchpad.net/maas/+bug/2130735

## Workaround
Finding the mpathX for that device, adding that disk
```bash
# ubuntu@msc-ostk-np-comp02:~$ sudo microceph disk add /dev/disk/by-id/dm-name-mpatha

+--------------------------------+---------+
| PATH | STATUS |
+--------------------------------+---------+
| /dev/disk/by-id/dm-name-mpatha | Success |
+--------------------------------+---------+
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.