canonical / canonical/microceph
Failed to add OSD disk
- Dominant language
- Go
- Stars
- 396
- Forks
- 74
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 7
Description
In a sunbeam deployment, adding disk using the charm failed with following error:
```
2025-09-11 23:30:19 DEBUG unit.microceph/0.juju-log server.go:406 Emitting Juju event add_osd_action.
2025-09-11 23:32:19 ERROR unit.microceph/0.juju-log server.go:406 Failed executing cmd: ['microceph', 'disk', 'add', '/dev/disk/by-dname/cajal.maas-disk'], error: Error: failed to request disk addition Post "http://control.socket/1.0/disks": context deadline exceeded
2025-09-11 23:32:19 ERROR unit.microceph/0.juju-log server.go:406 Error: failed to request disk addition Post "http://control.socket/1.0/disks": context deadline exceeded
```
Looking at the microceph snap logs
```
Sep 11 23:30:19 cajal microceph.daemon[57586]: time=2025-09-11T23:30:19.283Z level=INFO msg="Stabilizing device path for /dev/disk/by-dname/cajal.maas-disk"
Sep 11 23:30:19 cajal microceph.daemon[57586]: time=2025-09-11T23:30:19.308Z level=INFO msg="Set stable path for to /dev/disk/by-dname/cajal.maas-disk"
Sep 11 23:30:19 cajal microceph.daemon[57586]: time=2025-09-11T23:30:19.308Z level=INFO msg="Stabilized device path for /dev/disk/by-dname/cajal.maas-disk"
Sep 11 23:30:19 cajal microceph.daemon[57586]: time=2025-09-11T23:30:19.311Z level=INFO msg="Created disk record for osd.1"
Sep 11 23:30:19 cajal microceph.daemon[57586]: time=2025-09-11T23:30:19.311Z level=INFO msg="osd data path: /var/snap/microceph/common/data/osd/ceph-1"
Sep 11 23:30:20 cajal microceph.daemon[57586]: time=2025-09-11T23:30:20.111Z level=INFO msg="Generated OSD files for osd.1, fsid 3c62dfff-5c86-4c5c-961d-410bceedea96"
Sep 11 23:30:20 cajal microceph.daemon[57586]: time=2025-09-11T23:30:20.111Z level=INFO msg="Bootstrapping OSD /var/snap/microceph/common/data/osd/ceph-1 to 1"
Sep 11 23:31:24 cajal kernel: audit: type=1326 audit(1757633484.123:792): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=snap.microceph.daemon pid=63267 comm="ceph-osd" exe="/snap/microceph/1525/bin/ceph-osd" sig=0 arch=c000003e syscall=425 compat=0 ip=0x76e39a67225d code=0x50000
Sep 11 23:34:27 cajal microceph.daemon[57586]: time=2025-09-11T23:34:27.886Z level=INFO msg="OSD /var/snap/microceph/common/data/osd/ceph-1 bootstrapped successfully"
Sep 11 23:34:27 cajal microceph.daemon[57586]: time=2025-09-11T23:34:27.886Z level=INFO msg="Spawning OSD 1"
Sep 11 23:34:28 cajal microceph.daemon[57586]: time=2025-09-11T23:34:28.308Z level=INFO msg="Checking if we need to update failure domain for OSDs"
Sep 11 23:34:28 cajal microceph.daemon[57586]: time=2025-09-11T23:34:28.308Z level=ERROR msg="failed to update failure domain after adding OSD 1: failed to count members: Failed to begin transaction: context canceled"
Sep 11 23:34:28 cajal microceph.daemon[57586]: time=2025-09-11T23:34:28.310Z level=ERROR msg="failed to add disk: path /dev/disk/by-dname/cajal.maas-disk, err failed to count members: Failed to begin transaction: context canceled"
```
Versions:
charm-microceph squid/candidate 159
microceph snap 19.2.1+snap9d13f4a172 1525 squid/candidate
Contributor guide
Research direction
Start with the `microceph disk add` request and the microceph daemon path shown in the logs. Trace the disk-add operation from the HTTP request through OSD bootstrapping and failure-domain updates, using the reported context deadline and transaction errors as checkpoints. Done means the disk-add action completes successfully without these errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100