canonical / canonical/microceph

Failed to add OSD disk

Open
#623 10 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
396
Forks
74
Avg merge
2d 20h
Merged PRs (30d)
7

Description

In a sunbeam deployment, adding disk using the charm failed with following error:

```
2025-09-11 23:30:19 DEBUG unit.microceph/0.juju-log server.go:406 Emitting Juju event add_osd_action.
2025-09-11 23:32:19 ERROR unit.microceph/0.juju-log server.go:406 Failed executing cmd: ['microceph', 'disk', 'add', '/dev/disk/by-dname/cajal.maas-disk'], error: Error: failed to request disk addition Post "http://control.socket/1.0/disks": context deadline exceeded

2025-09-11 23:32:19 ERROR unit.microceph/0.juju-log server.go:406 Error: failed to request disk addition Post "http://control.socket/1.0/disks": context deadline exceeded

```

Looking at the microceph snap logs
```
Sep 11 23:30:19 cajal microceph.daemon[57586]: time=2025-09-11T23:30:19.283Z level=INFO msg="Stabilizing device path for /dev/disk/by-dname/cajal.maas-disk"
Sep 11 23:30:19 cajal microceph.daemon[57586]: time=2025-09-11T23:30:19.308Z level=INFO msg="Set stable path for to /dev/disk/by-dname/cajal.maas-disk"
Sep 11 23:30:19 cajal microceph.daemon[57586]: time=2025-09-11T23:30:19.308Z level=INFO msg="Stabilized device path for /dev/disk/by-dname/cajal.maas-disk"
Sep 11 23:30:19 cajal microceph.daemon[57586]: time=2025-09-11T23:30:19.311Z level=INFO msg="Created disk record for osd.1"
Sep 11 23:30:19 cajal microceph.daemon[57586]: time=2025-09-11T23:30:19.311Z level=INFO msg="osd data path: /var/snap/microceph/common/data/osd/ceph-1"
Sep 11 23:30:20 cajal microceph.daemon[57586]: time=2025-09-11T23:30:20.111Z level=INFO msg="Generated OSD files for osd.1, fsid 3c62dfff-5c86-4c5c-961d-410bceedea96"
Sep 11 23:30:20 cajal microceph.daemon[57586]: time=2025-09-11T23:30:20.111Z level=INFO msg="Bootstrapping OSD /var/snap/microceph/common/data/osd/ceph-1 to 1"
Sep 11 23:31:24 cajal kernel: audit: type=1326 audit(1757633484.123:792): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=snap.microceph.daemon pid=63267 comm="ceph-osd" exe="/snap/microceph/1525/bin/ceph-osd" sig=0 arch=c000003e syscall=425 compat=0 ip=0x76e39a67225d code=0x50000
Sep 11 23:34:27 cajal microceph.daemon[57586]: time=2025-09-11T23:34:27.886Z level=INFO msg="OSD /var/snap/microceph/common/data/osd/ceph-1 bootstrapped successfully"
Sep 11 23:34:27 cajal microceph.daemon[57586]: time=2025-09-11T23:34:27.886Z level=INFO msg="Spawning OSD 1"
Sep 11 23:34:28 cajal microceph.daemon[57586]: time=2025-09-11T23:34:28.308Z level=INFO msg="Checking if we need to update failure domain for OSDs"
Sep 11 23:34:28 cajal microceph.daemon[57586]: time=2025-09-11T23:34:28.308Z level=ERROR msg="failed to update failure domain after adding OSD 1: failed to count members: Failed to begin transaction: context canceled"
Sep 11 23:34:28 cajal microceph.daemon[57586]: time=2025-09-11T23:34:28.310Z level=ERROR msg="failed to add disk: path /dev/disk/by-dname/cajal.maas-disk, err failed to count members: Failed to begin transaction: context canceled"
```

Versions:
charm-microceph squid/candidate 159
microceph snap 19.2.1+snap9d13f4a172 1525 squid/candidate

Contributor guide

Open the contributing guide

Research direction

Start with the `microceph disk add` request and the microceph daemon path shown in the logs. Trace the disk-add operation from the HTTP request through OSD bootstrapping and failure-domain updates, using the reported context deadline and transaction errors as checkpoints. Done means the disk-add action completes successfully without these errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.