canonical / canonical/microceph

MicroCeph fails on machine with device mapper multipathing

Open
#375 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
396
Forks
74
Avg merge
2d 20h
Merged PRs (30d)
7

Description

# Issue report

## What version of MicroCeph are you using ?

```
$ snap info microceph
...
installed: 18.2.0+snapcba31e8c75 (999) 87MB -
```

## What are the steps to reproduce this issue ?

Using the following script to create LXD VMs:
```bash
set -x
n=0

for node in micro-1 micro-2 micro-3; do
n=$(( n + 1 ))
lxc stop microcloud:sunbeam-$n || true
lxc delete microcloud:sunbeam-$n || true
lxc storage volume delete microcloud:default sunbeam-$n-disk-2 || true
lxc storage volume delete microcloud:default sunbeam-$n-disk-3 || true
lxc storage volume delete microcloud:default sunbeam-$n-disk-4 || true

lxc launch ubuntu:jammy microcloud:sunbeam-$n \
--vm \
-c limits.cpu=16 \
-c limits.memory=32GiB \
--target $node
lxc storage volume create microcloud:default \
sunbeam-$n-disk-2 size=100GiB --type=block --target=$node
lxc storage volume create microcloud:default \
sunbeam-$n-disk-3 size=100GiB --type=block --target=$node
lxc storage volume create microcloud:default \
sunbeam-$n-disk-4 size=100GiB --type=block --target=$node

lxc config device add microcloud:sunbeam-$n eth1 nic network=internal
lxc config device add microcloud:sunbeam-$n \
sunbeam-$n-disk-2 disk pool=default source=sunbeam-$n-disk-2
lxc config device add microcloud:sunbeam-$n \
sunbeam-$n-disk-3 disk pool=default source=sunbeam-$n-disk-3
lxc config device add microcloud:sunbeam-$n \
sunbeam-$n-disk-4 disk pool=default source=sunbeam-$n-disk-4
done

```

## What happens (observed behaviour) ?

```
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 63.9M 1 loop /snap/core20/2318
loop1 7:1 0 38.8M 1 loop /snap/snapd/21759
loop2 7:2 0 87M 1 loop /snap/lxd/28373
sda 8:0 0 100G 0 disk
├─sda1 8:1 0 99.9G 0 part /
├─sda14 8:14 0 4M 0 part
└─sda15 8:15 0 106M 0 part /boot/efi
sdb 8:16 0 100G 0 disk
└─mpatha 253:0 0 100G 0 mpath
sdc 8:32 0 100G 0 disk
└─mpatha 253:0 0 100G 0 mpath
sdd 8:48 0 100G 0 disk
└─mpatha 253:0 0 100G 0 mpath

$ sudo microceph disk list

Available unpartitioned disks on this system:
+---------------+-----------+------+------------------------------------------------------------------+
| MODEL | CAPACITY | TYPE | PATH |
+---------------+-----------+------+------------------------------------------------------------------+
| QEMU HARDDISK | 100.00GiB | scsi | /dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_lxd_sunbeam--1--disk--2 |
+---------------+-----------+------+------------------------------------------------------------------+
| QEMU HARDDISK | 100.00GiB | scsi | /dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_lxd_sunbeam--1--disk--3 |
+---------------+-----------+------+------------------------------------------------------------------+
| QEMU HARDDISK | 100.00GiB | scsi | /dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_lxd_sunbeam--1--disk--4 |
+---------------+-----------+------+------------------------------------------------------------------+

$ sudo microceph disk add /dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_lxd_sunbeam--1--disk--2

+------------------------------------------------------------------+---------+
| PATH | STATUS |
+------------------------------------------------------------------+---------+
| /dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_lxd_sunbeam--1--disk--2 | Failure |
+------------------------------------------------------------------+---------+
Error: failed to bootstrap OSD: Failed to run: ceph-osd --mkfs --no-mon-config -i 1: exit status 250 (2024-07-02T09:34:17.422+0000 7f2e824d58c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input
2024-07-02T09:34:17.422+0000 7f2e824d58c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input
2024-07-02T09:34:17.422+0000 7f2e824d58c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input
2024-07-02T09:34:17.450+0000 7f2e824d58c0 -1 bdev(0x55735efae000 /var/lib/ceph/osd/ceph-1/block) open open got: (16) Device or resource busy
2024-07-02T09:34:17.450+0000 7f2e824d58c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) mkfs failed, (16) Device or resource busy
2024-07-02T09:34:17.450+0000 7f2e824d58c0 -1 OSD::mkfs: ObjectStore::mkfs failed with error (16) Device or resource busy
2024-07-02T09:34:17.450+0000 7f2e824d58c0 -1 ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-1: (16) Device or resource busy)
```

## What were you expecting to happen ?

Successful addition of disk.

## Relevant logs, error output, etc.

See above.

## Additional comments.

Disabling multipath with `sudo multipath -F` allows disks to be added.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.