canonical / canonical/microceph
osd.start `set -eu` prevents LUKS unlock of subsequent OSDs when one OSD fails to start
- Dominant language
- Go
- Stars
- 396
- Forks
- 74
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 7
Description
# Bug Report: osd.start `set -eu` prevents LUKS unlock of subsequent OSDs when one OSD fails to start
## Summary
The `snapcraft/commands/osd.start` script uses `set -eu` and runs `ceph-osd` in the foreground for each OSD in alphabetical order. If one OSD fails to start (e.g., bluestore corruption), the script exits immediately due to `set -e`, preventing any subsequent OSDs from being processed — including their LUKS unlock step. This leaves later OSDs permanently down after a reboot until manually unlocked.
## Environment
- **MicroCeph Version:** 19.2.3+snapafb6a3a0cb (squid/stable, snap revision 1641)
- **MicroCloud:** 2.1.2-9cfe3d5
- **OS:** Ubuntu 24.04.4 LTS (Noble Numbat)
- **Kernel:** 6.8.0-136-generic
- **Storage:** 3x Samsung MZQL2960HCJR 894GB NVMe SSDs
- osd.8: nvme1n1 (LUKS encrypted)
- osd.9: nvme2n1 (LUKS encrypted)
- osd.10: nvme0n1 (unencrypted)
## Steps to Reproduce
1. Deploy a MicroCeph cluster with 3 OSDs, where osd.8 and osd.9 use LUKS encryption and osd.10 does not.
2. Introduce a bluestore corruption on osd.8 (e.g., via an unclean shutdown while writing).
3. Reboot the node.
4. Observe that:
- osd.10 starts successfully (processed first alphabetically: `ceph-10`)
- osd.8 is unlocked successfully by `maybe_unlock()`, but `ceph-osd --id 8` fails with bluestore fsck error → script exits due to `set -eu`
- osd.9 is **never reached** in the loop → `luksosd-9` mapper device is never created → osd.9 stays down permanently
## Root Cause
In `snapcraft/commands/osd.start`:
```bash
#!/bin/bash
set -eu # ← exits on ANY error
spawn() {
for i in "${SNAP_COMMON}/data/osd"/*; do
# ... (alphabetical order: ceph-10, ceph-8, ceph-9)
if [ -b "${i}/unencrypted" ] ; then
maybe_unlock "${i}/unencrypted" "${nr}" "$( get_key "${nr}" )"
fi
ceph-osd --cluster ceph --id "${nr}" # ← runs in FOREGROUND, blocks
# If this exits non-zero, set -e kills the entire script
done
}
```
The script iterates OSD directories alphabetically (`ceph-10`, `ceph-8`, `ceph-9`) and runs `ceph-osd` in the foreground for each. When `ceph-osd` fails (e.g., bluestore corruption), `set -eu` causes the script to exit before it reaches subsequent OSDs in the iteration. Those OSDs never get their `maybe_unlock()` called, so their LUKS devices remain locked.
This is a different bug from #655 (which was about WAL/DB devices not being unlocked). Here, the issue is that the entire `maybe_unlock` step is skipped for OSDs that come after a failing OSD in the iteration order.
## Observed Behavior
After a reboot with osd.8 having bluestore corruption:
```
$ sudo ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS
8 ssd 0.87329 osd.8 down ← bluestore corruption (expected)
9 ssd 0.87329 osd.9 down ← NOT unlocked, never reached by script
10 ssd 0.87328 osd.10 up ← started first (alphabetical: ceph-10)
$ sudo dmsetup ls
luksosd-8 (252:0) ← unlocked by maybe_unlock before osd.8 failed
← luksosd-9 MISSING: never created
```
OSD service crash-loops because osd.8 fails, but osd.9's LUKS device is never created:
```
Jul 22 20:06:47 vds1 microceph.osd[1539132]: bluestore(/var/lib/ceph/osd/ceph-8) fsck error: stray spanning blob found:27
Jul 22 20:07:01 vds1 microceph.osd[1539132]: osd.8 0 OSD:init: unable to mount object store
Jul 22 20:07:01 vds1 microceph.osd[1539132]: ** ERROR: osd init failed: (5) Input/output error
Jul 22 20:07:02 vds1 systemd[1]: snap.microceph.osd.service: Main process exited, code=exited, status=250/n/a
```
## Expected Behavior
All LUKS-encrypted OSDs should be unlocked before any `ceph-osd` is started, or the script should continue to the next OSD when one fails, rather than exiting entirely.
## Proposed Fix
### Option A: Separate unlock pass from spawn pass
Unlock all LUKS devices first, then start OSDs with error tolerance:
```bash
spawn() {
# Pass 1: Unlock all LUKS devices
for i in "${SNAP_COMMON}/data/osd"/*; do
filename="$(basename "${i}")"
[ -z "$filename" ] && continue
nr="${filename##ceph-}"
[ -z "$nr" ] && continue
[ ! -e "${i}/ready" ] && continue
if [ -b "${i}/unencrypted" ] ; then
maybe_unlock "${i}/unencrypted" "${nr}" "$( get_key "${nr}" )" || true
fi
done
# Pass 2: Start OSDs (don't exit on single OSD failure)
for i in "${SNAP_COMMON}/data/osd"/*; do
filename="$(basename "${i}")"
[ -z "$filename" ] && continue
nr="${filename##ceph-}"
[ -z "$nr" ] && continue
[ ! -e "${i}/ready" ] && continue
is_osd_running "${nr}" && continue
ceph-osd --cluster ceph --id "${nr}" &
done
wait
sleep infinity &
wait
}
```
### Option B: Background each ceph-osd and use `|| true`
```bash
spawn() {
for i in "${SNAP_COMMON}/data/osd"/*; do
# ... existing checks ...
if [ -b "${i}/unencrypted" ] ; then
maybe_unlock "${i}/unencrypted" "${nr}" "$( get_key "${nr}" )"
fi
ceph-osd --cluster ceph --id "${nr}" & # background, don't block
done
wait
sleep infinity &
wait
}
```
## Workaround
Add LUKS entries to `/etc/crypttab` so the OS unlocks all encrypted OSD devices at boot, before the MicroCeph OSD service starts. See: [MicroCeph LUKS auto-unlock fix](https://github.com/canonical/microceph/issues/655#issuecomment-...)
## Related Issues
- #655 — OSD fails to start after reboot if using encrypted WAL or DB devices (same file, different bug — WAL/DB unlock was missing)
- #34 — Add support for LUKS encryption
Contributor guide
Research direction
Start with snapcraft/commands/osd.start, especially spawn(), maybe_unlock(), and the foreground ceph-osd call. Reproduce the alphabetical OSD failure scenario or inspect the existing startup flow, then verify that every eligible encrypted OSD is unlocked and later OSDs are still processed when one ceph-osd fails.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- bash
- Domain
- distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100