canonical / canonical/microceph

osd.start `set -eu` prevents LUKS unlock of subsequent OSDs when one OSD fails to start

Open
#808 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
396
Forks
74
Avg merge
2d 20h
Merged PRs (30d)
7

Description

# Bug Report: osd.start `set -eu` prevents LUKS unlock of subsequent OSDs when one OSD fails to start

## Summary

The `snapcraft/commands/osd.start` script uses `set -eu` and runs `ceph-osd` in the foreground for each OSD in alphabetical order. If one OSD fails to start (e.g., bluestore corruption), the script exits immediately due to `set -e`, preventing any subsequent OSDs from being processed — including their LUKS unlock step. This leaves later OSDs permanently down after a reboot until manually unlocked.

## Environment

- **MicroCeph Version:** 19.2.3+snapafb6a3a0cb (squid/stable, snap revision 1641)
- **MicroCloud:** 2.1.2-9cfe3d5
- **OS:** Ubuntu 24.04.4 LTS (Noble Numbat)
- **Kernel:** 6.8.0-136-generic
- **Storage:** 3x Samsung MZQL2960HCJR 894GB NVMe SSDs
- osd.8: nvme1n1 (LUKS encrypted)
- osd.9: nvme2n1 (LUKS encrypted)
- osd.10: nvme0n1 (unencrypted)

## Steps to Reproduce

1. Deploy a MicroCeph cluster with 3 OSDs, where osd.8 and osd.9 use LUKS encryption and osd.10 does not.
2. Introduce a bluestore corruption on osd.8 (e.g., via an unclean shutdown while writing).
3. Reboot the node.
4. Observe that:
- osd.10 starts successfully (processed first alphabetically: `ceph-10`)
- osd.8 is unlocked successfully by `maybe_unlock()`, but `ceph-osd --id 8` fails with bluestore fsck error → script exits due to `set -eu`
- osd.9 is **never reached** in the loop → `luksosd-9` mapper device is never created → osd.9 stays down permanently

## Root Cause

In `snapcraft/commands/osd.start`:

```bash
#!/bin/bash
set -eu # ← exits on ANY error

spawn() {
for i in "${SNAP_COMMON}/data/osd"/*; do
# ... (alphabetical order: ceph-10, ceph-8, ceph-9)

if [ -b "${i}/unencrypted" ] ; then
maybe_unlock "${i}/unencrypted" "${nr}" "$( get_key "${nr}" )"
fi

ceph-osd --cluster ceph --id "${nr}" # ← runs in FOREGROUND, blocks
# If this exits non-zero, set -e kills the entire script
done
}
```

The script iterates OSD directories alphabetically (`ceph-10`, `ceph-8`, `ceph-9`) and runs `ceph-osd` in the foreground for each. When `ceph-osd` fails (e.g., bluestore corruption), `set -eu` causes the script to exit before it reaches subsequent OSDs in the iteration. Those OSDs never get their `maybe_unlock()` called, so their LUKS devices remain locked.

This is a different bug from #655 (which was about WAL/DB devices not being unlocked). Here, the issue is that the entire `maybe_unlock` step is skipped for OSDs that come after a failing OSD in the iteration order.

## Observed Behavior

After a reboot with osd.8 having bluestore corruption:

```
$ sudo ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS
8 ssd 0.87329 osd.8 down ← bluestore corruption (expected)
9 ssd 0.87329 osd.9 down ← NOT unlocked, never reached by script
10 ssd 0.87328 osd.10 up ← started first (alphabetical: ceph-10)

$ sudo dmsetup ls
luksosd-8 (252:0) ← unlocked by maybe_unlock before osd.8 failed
← luksosd-9 MISSING: never created
```

OSD service crash-loops because osd.8 fails, but osd.9's LUKS device is never created:
```
Jul 22 20:06:47 vds1 microceph.osd[1539132]: bluestore(/var/lib/ceph/osd/ceph-8) fsck error: stray spanning blob found:27
Jul 22 20:07:01 vds1 microceph.osd[1539132]: osd.8 0 OSD:init: unable to mount object store
Jul 22 20:07:01 vds1 microceph.osd[1539132]: ** ERROR: osd init failed: (5) Input/output error
Jul 22 20:07:02 vds1 systemd[1]: snap.microceph.osd.service: Main process exited, code=exited, status=250/n/a
```

## Expected Behavior

All LUKS-encrypted OSDs should be unlocked before any `ceph-osd` is started, or the script should continue to the next OSD when one fails, rather than exiting entirely.

## Proposed Fix

### Option A: Separate unlock pass from spawn pass

Unlock all LUKS devices first, then start OSDs with error tolerance:

```bash
spawn() {
# Pass 1: Unlock all LUKS devices
for i in "${SNAP_COMMON}/data/osd"/*; do
filename="$(basename "${i}")"
[ -z "$filename" ] && continue
nr="${filename##ceph-}"
[ -z "$nr" ] && continue
[ ! -e "${i}/ready" ] && continue

if [ -b "${i}/unencrypted" ] ; then
maybe_unlock "${i}/unencrypted" "${nr}" "$( get_key "${nr}" )" || true
fi
done

# Pass 2: Start OSDs (don't exit on single OSD failure)
for i in "${SNAP_COMMON}/data/osd"/*; do
filename="$(basename "${i}")"
[ -z "$filename" ] && continue
nr="${filename##ceph-}"
[ -z "$nr" ] && continue
[ ! -e "${i}/ready" ] && continue
is_osd_running "${nr}" && continue

ceph-osd --cluster ceph --id "${nr}" &
done

wait
sleep infinity &
wait
}
```

### Option B: Background each ceph-osd and use `|| true`

```bash
spawn() {
for i in "${SNAP_COMMON}/data/osd"/*; do
# ... existing checks ...
if [ -b "${i}/unencrypted" ] ; then
maybe_unlock "${i}/unencrypted" "${nr}" "$( get_key "${nr}" )"
fi
ceph-osd --cluster ceph --id "${nr}" & # background, don't block
done
wait
sleep infinity &
wait
}
```

## Workaround

Add LUKS entries to `/etc/crypttab` so the OS unlocks all encrypted OSD devices at boot, before the MicroCeph OSD service starts. See: [MicroCeph LUKS auto-unlock fix](https://github.com/canonical/microceph/issues/655#issuecomment-...)

## Related Issues

- #655 — OSD fails to start after reboot if using encrypted WAL or DB devices (same file, different bug — WAL/DB unlock was missing)
- #34 — Add support for LUKS encryption

Contributor guide

Open the contributing guide

Research direction

Start with snapcraft/commands/osd.start, especially spawn(), maybe_unlock(), and the foreground ceph-osd call. Reproduce the alphabetical OSD failure scenario or inspect the existing startup flow, then verify that every eligible encrypted OSD is unlocked and later OSDs are still processed when one ceph-osd fails.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash
Domain
distributed-systems, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.