OSDs on squid/stable (19.2.3) crash-loop

# OSDs on squid/stable (19.2.3) crash-loop with `BlueFS::truncate` assert `ceph_assert(cut_off == p->length)`

## Summary

On a MicroCeph cluster (deployed as part of a MicroCloud) running `19.2.3+snapcf306793a4` (squid/stable), two OSDs on separate hosts became permanently unstartable. Both crash-loop on startup with an identical BlueStore/BlueFS assertion during RocksDB WAL recovery:

```
./src/os/bluestore/BlueFS.cc: 3871: FAILED ceph_assert(cut_off == p->length)
```

`ceph-bluestore-tool fsck` (read-only) completes successfully, while `repair` (read-write) aborts at the same assert. The OSDs could not be recovered in place and were rebuilt.

This appears to correspond to upstream Ceph tracker #71235 [\[1\]][r1] ("Adapt BlueFS::truncate to changing allocation unit"). The upstream fix is pr#63753 [\[2\]][r2] ("os/bluestore: In BlueFS::truncate accept wierd alloc_unit"), which is listed in the changelog of the Ceph v19.2.4 (Squid) release [\[3\]][r3], described there as "the fourth backport release in the Squid series."

At the time of observation, the MicroCeph `squid` track packages Ceph `19.2.3` across all published revisions; `19.2.4` (which contains pr#63753 [\[2\]][r2]) does not appear on the squid channel. This issue is filed to document the observed failure and the version details, in case it is useful for tracking snap/track alignment with upstream Squid point releases.

## Environment

| Field | Value |
|---|---|
| Snap | microceph |
| Installed version | `19.2.3+snapcf306793a4` |
| Installed revision | 1735 |
| Channel | squid/stable |
| Ceph version string | `19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable)` |
| Deployment | MicroCloud (MicroCeph providing storage for MicroCloud VMs) |
| Cluster | 6 hosts, 6 OSDs, replicated pools (size 3, min_size 2) |
| OSD backing | Physical block devices |

### Observed squid/stable channel state (store API)

Query used:
```
curl -sH 'Snap-Device-Series: 16' \
'https://api.snapcraft.io/v2/snaps/info/microceph' \
| jq -r '.["channel-map"][] | select(.channel.name=="squid/stable") | .version, .revision'
```

Output — every published squid/stable revision reports Ceph 19.2.3:
```
19.2.3+snapcf306793a4 1735
19.2.3+snapcf306793a4 1736
19.2.3+snapcf306793a4 1737
19.2.3+snapcf306793a4 1738
19.2.3+snapcf306793a4 1740
```

For reference, `snap info microceph` at the time also showed `tentacle/stable` on `20.2.1` (a major version ahead of Squid) and no `19.2.4` on any squid channel.

## Observed sequence

1. The cluster (a 6-host MicroCloud deployment with 6 OSDs) was healthy prior to the event, with all 6 OSDs up.
2. Two OSDs on separate hosts (`osd.3` on host A, `osd.4` on host B) went down within the same window and then failed to restart, leaving the cluster at 4 of 6 OSDs up.
3. The direct cause of the initial outage could not be verified. There was no host reboot or power event — on the host carrying `osd.4`, `uptime -s`, `last -x`, and `journalctl --list-boots` confirm the host ran continuously across the OSD-failure window, and no crash entry was recorded (`ceph crash ls` empty). The working theory is I/O saturation: during later recovery, other OSDs (`osd.2`, `osd.5`, `osd.6`) were observed dropping under load with `bdev ... aio_submit retries` messages and returning cleanly on restart, so the same mechanism may have marked `osd.3`/`osd.4` down. This is unverified for the original event, as logs from the moment of first failure were not captured.
4. Both failed OSDs then crash-loop on restart with the BlueFS `cut_off == p->length` assert on mount, within the RocksDB WAL recovery path, and could not be recovered in place.
5. At the time of failure the OSDs were fairly full (~90% used) with fragmentation — the high-fill condition described in community reports of this assert [\[5\]][r5].
6. With two OSDs down and the remaining OSDs near capacity, re-replication hit `backfill_toofull`; 16 PGs became inactive (`undersized+degraded+remapped+backfill_toofull+peered`), which blocked RBD I/O to the MicroCloud VMs until enough OSDs were brought back to activate the affected PGs.

## Crash backtrace (OSD startup)

```
./src/os/bluestore/BlueFS.cc: In function 'int BlueFS::truncate(FileWriter*, uint64_t)'
./src/os/bluestore/BlueFS.cc: 3871: FAILED ceph_assert(cut_off == p->length)
ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable)
...
3: (BlueFS::truncate(BlueFS::FileWriter*, unsigned long)+0x8b7)
4: (BlueRocksWritableFile::Close()+0x35)
6: (rocksdb::WritableFileWriter::Close()+0x803)
7: (rocksdb::BuildTable(...)+0x2a45)
8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(...)+0x1019)
9: (rocksdb::DBImpl::RecoverLogFiles(...)+0x1db7)
10: (rocksdb::DBImpl::Recover(...)+0x18b8)
11: (rocksdb::DBImpl::Open(...)+0x75b)
13: (RocksDBStore::do_open(...)+0xae5)
14: (BlueStore::_open_db(bool, bool, bool)+0x806)
15: (BlueStore::_open_db_and_around(bool, bool)+0x48a)
16: (BlueStore::_mount()+0x23f)
17: (OSD::init()+0x4e6)
```

## Diagnostic observations

- `ceph-bluestore-tool fsck --path /var/lib/ceph/osd/ceph-` completes successfully (RocksDB opened read-only; the failing write path is not exercised).
- `ceph-bluestore-tool fsck --path ... --bluefs_replay_recovery=true --bluefs_replay_recovery_disable_compact=true` also completes successfully.
- `ceph-bluestore-tool repair --path /var/lib/ceph/osd/ceph-` aborts with the same `cut_off == p->length` assert. The crash is in the WAL-flush write path (`WriteLevel0TableForRecovery` → `BuildTable` → `BlueFS::truncate`).
- Underlying disks show no hardware errors: no `dmesg` I/O errors and no SMART failures.

## Impact observed

Cluster-wide RBD I/O stall (MicroCloud VMs unresponsive) resulting from 16 inactive PGs. In-place recovery of the affected OSDs was not possible on 19.2.3; the data was recovered only because other OSDs retained a surviving copy of the affected PGs.

## Additional notes

- On 19.2.3, `ceph-bluestore-tool repair` did not resolve the condition (the upstream fix [\[2\]][r2] is a code change rather than a data repair).
- A start-time workaround has been reported by others (reducing the BlueStore shared allocation size to 4096 so the OSD can start despite unaligned free space); it was not applied here.
- Keeping OSD utilization well below ~75% appears to reduce the likelihood of encountering the assert on 19.2.3, based on community reports correlating it with high fill and fragmentation [\[5\]][r5].
- A change in 19.2.3 that may be related to the regression is `os/bluestore: use block size (4K) as minimal allocation unit for dedicated DB/WAL volumes` (pr#62514), per the 19.2.3 release notes [\[4\]][r4].

## References

1. Ceph tracker #71235 — "Adapt BlueFS::truncate to changing allocation unit": [tracker.ceph.com/issues/71235](https://tracker.ceph.com/issues/71235)
2. Ceph pr#63753 — "os/bluestore: In BlueFS::truncate accept wierd alloc_unit": [github.com/ceph/ceph/pull/63753](https://github.com/ceph/ceph/pull/63753)
3. Ceph v19.2.4 (Squid) release notes (lists pr#63753): [ceph.io/en/news/blog/2026/v19-2-4-squid-released](https://ceph.io/en/news/blog/2026/v19-2-4-squid-released/)
4. Ceph v19.2.3 (Squid) release notes: [ceph.io/en/news/blog/2025/v19-2-3-squid-released](https://ceph.io/en/news/blog/2025/v19-2-3-squid-released/)
5. Clyso — "Critical bugs in Ceph Reef and Squid": [docs.clyso.com/blog/critical-bugs-ceph-reef-squid](https://docs.clyso.com/blog/critical-bugs-ceph-reef-squid/)

[r1]: https://tracker.ceph.com/issues/71235
[r2]: https://github.com/ceph/ceph/pull/63753
[r3]: https://ceph.io/en/news/blog/2026/v19-2-4-squid-released/
[r4]: https://ceph.io/en/news/blog/2025/v19-2-3-squid-released/
[r5]: https://docs.clyso.com/blog/critical-bugs-ceph-reef-squid/

## AI Use Disclosure
Report drafted with AI assistance; all diagnostics are from a live cluster and independently verified.

No MicroCeph source file, test, or entry point is named. Start by reviewing the squid/stable channel versions and the referenced Ceph tracker and upstream pull request; the issue is resolved when the packaging or release-tracking discrepancy is addressed, but no specific repository change is defined here.

Written by the indexing model from the issue text.