canonical / canonical/microceph

osds flipping (OSDs going up and down)

Open
#614 9 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
396
Forks
74
Avg merge
2d 20h
Merged PRs (30d)
7

Description

Hi Team,
I need help with my single node microceph installation.
I have three osds.
2 sdd + 1 loop file.

All ssd are flipping every few mins.

Load is in kb.
running:
**ceph-version: 19.2.0-0ubuntu0.24.04.2; microceph-git: 2fbf0bad05**

**OSD:**
one disk is nvme (M2).
2nd one is SSD
3rd is loop file (also on SSD)

```
[22779.472290] libceph: osd2 (1)[REDACTED]:6813 socket error on write
[22780.395746] audit: type=1326 audit(1755520067.795:4532): auid=[REDACTED] uid=0 gid=0 ses=[REDACTED] subj=snap.microceph.osd pid=3694705 comm="ceph-osd" exe="/snap/microceph/[REDACTED]/bin/ceph-osd" sig=0 arch=c000003e syscall=425 compat=0 ip=0x766730bac25d code=0x50000
[22784.209375] libceph ([REDACTED] e12774): osd10 up
[22784.286550] audit: type=1326 audit(1755520071.686:4533): auid=[REDACTED] uid=0 gid=0 ses=[REDACTED] subj=snap.microceph.osd pid=3695303 comm="ceph-osd" exe="/snap/microceph/[REDACTED]/bin/ceph-osd" sig=0 arch=c000003e syscall=425 compat=0 ip=0x7d2e5093425d code=0x50000
[22785.030020] libceph: wrong peer, want (1)[REDACTED]:6813/[REDACTED], got (1)[REDACTED]:6813/[REDACTED]
[22785.030025] libceph: osd2 (1)[REDACTED]:6813 wrong peer at address
[22786.882592] libceph: osd10 (1)[REDACTED]:6803 socket closed (con state OPEN)
[22786.882713] libceph: osd10 (1)[REDACTED]:6803 socket error on write
[22786.883028] libceph ([REDACTED] e12777): osd2 down
[22786.883032] libceph ([REDACTED] e12777): osd10 down
[22791.620083] audit: type=1326 audit(1755520079.019:4534): auid=[REDACTED] uid=0 gid=0 ses=[REDACTED] subj=snap.microceph.osd pid=3696037 comm="ceph-osd" exe="/snap/microceph/[REDACTED]/bin/ceph-osd" sig=0 arch=c000003e syscall=425 compat=0 ip=0x7b655383b25d code=0x50000
[22794.456001] libceph ([REDACTED] e12779): osd10 up
[22796.423508] audit: type=1326 audit(1755520083.822:4535): auid=[REDACTED] uid=0 gid=0 ses=[REDACTED] subj=snap.microceph.osd pid=3696628 comm="ceph-osd" exe="/snap/microceph/[REDACTED]/bin/ceph-osd" sig=0 arch=c000003e syscall=425 compat=0 ip=0x7175fb6d325d code=0x50000
[22799.570105] libceph ([REDACTED] e12782): osd10 down
[22799.570123] libceph ([REDACTED] e12782): osd2 up
[22799.570604] libceph: osd2 (1)[REDACTED]:6813 socket error on write
[22799.826587] libceph: osd2 (1)[REDACTED]:6813 socket error on write
[22800.328615] libceph: osd2 (1)[REDACTED]:6813 socket error on write
[22801.490758] libceph: osd2 (1)[REDACTED]:6813 socket error on write
[22803.473806] libceph: osd2 (1)[REDACTED]:6813 socket error on write
[22804.386275] audit: type=1326 audit(1755520091.784:4536): auid=[REDACTED] uid=0 gid=0 ses=[REDACTED] subj=snap.microceph.osd pid=3697363 comm="ceph-osd" exe="/snap/microceph/[REDACTED]/bin/ceph-osd" sig=0 arch=c000003e syscall=425 compat=0 ip=0x7c6f7ef9925d code=0x50000
[22809.284237] audit: type=1326 audit(1755520096.682:4537): auid=[REDACTED] uid=0 gid=0 ses=[REDACTED] subj=snap.microceph.osd pid=3697961 comm="ceph-osd" exe="/snap/microceph/[REDACTED]/bin/ceph-osd" sig=0 arch=c000003e syscall=425 compat=0 ip=0x7af6d523325d code=0x50000
[22809.811121] libceph ([REDACTED] e12784): osd10 up
[22810.340393] libceph: wrong peer, want (1)[REDACTED]:6813/[REDACTED], got (1)[REDACTED]:6813/[REDACTED]
[22810.340412] libceph: osd2 (1)[REDACTED]:6813 wrong peer at address
[22812.003062] libceph: osd10 (1)[REDACTED]:6803 socket closed (con state OPEN)
[22812.003166] libceph: osd10 (1)[REDACTED]:6803 socket error on write
[22812.003453] libceph ([REDACTED] e12787): osd2 down
[22812.003456] libceph ([REDACTED] e12787): osd10 down
[22816.582845] audit: type=1326 audit(1755520103.982:4538): auid=[REDACTED] uid=0 gid=0 ses=[REDACTED] subj=snap.microceph.osd pid=3698695 comm="ceph-osd" exe="/snap/microceph/[REDACTED]/bin/ceph-osd" sig=0 arch=c000003e syscall=425 compat=0 ip=0x7192ee48625d code=0x50000

```

systemctl status snap.microceph.osd.service
```
● snap.microceph.osd.service - Service for snap application microceph.osd
Loaded: loaded (/etc/systemd/system/snap.microceph.osd.service; enabled; preset: enabled)
Active: active (running) since Tue 2025-08-19 11:50:43 CEST; 493ms ago
Main PID: 4178795 (osd.start)
Tasks: 2 (limit: 38205)
Memory: 700.0K (peak: 12.9M)
CPU: 56ms
CGroup: /system.slice/snap.microceph.osd.service
├─4178795 /bin/sh /snap/microceph/1271/commands/osd.start
└─4178830 sleep 1

Aug 19 11:50:43 antaresinc-cluster systemd[1]: snap.microceph.osd.service: Scheduled restart job, restart counter is at 15.
Aug 19 11:50:43 antaresinc-cluster systemd[1]: Started snap.microceph.osd.service - Service for snap application microceph.osd.
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.