canonical / canonical/microceph

RGW admin datalog request segfaults radosgw (upstream Ceph 20.2.1)

Open
#810 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
396
Forks
74
Avg merge
2d 20h
Merged PRs (30d)
7

Description

## Summary

On the `tentacle/candidate` snap channel, `radosgw` crashes (SIGSEGV) handling a routine admin HTTP request. RGW (RADOS Gateway) multisite data sync issues this request automatically and continuously, so any two `tentacle/candidate` clusters configured for RGW multisite crash-loop `radosgw` and object data never replicates. Metadata sync is unaffected.

This is an **upstream Ceph 20.2.1 defect**, not a MicroCeph code issue — see "Root cause" below — but it currently makes `tentacle/candidate` unusable for RGW replication work (blocking CE062).

## Environment

```
$ snap info microceph | grep candidate
$ microceph.radosgw-admin --version
ceph version 20.2.1 (6a49aff47758778a5f5951e731d437c317f72fb2) tentacle (stable)
```
Two single-node clusters, LXD containers, `microceph disk add loop,4G,3` (3 OSDs each), both `HEALTH_OK`.

## Steps to reproduce

Three scripts, each self-contained (run inside a fresh Ubuntu 24.04 host/VM with LXD and internet access), all tested and attached alongside this report:

| Script | What it shows |
|---|---|
| [reproduce-microceph-minimal.sh](https://github.com/user-attachments/files/30326991/reproduce-microceph-minimal.sh) | One MicroCeph cluster, no multisite, no peer zone, no user data. Enables RGW and sends one signed admin request that crashes it. Fastest way to confirm the bug. |
| [reproduce-microceph-multisite.sh](https://github.com/user-attachments/files/30326992/reproduce-microceph-multisite.sh) | The real-world scenario: two MicroCeph clusters paired for RGW multisite. No manual trigger needed — data sync polls the crashing endpoint on its own every ~30s. |
| [reproduce-vanilla-packages.sh](https://github.com/user-attachments/files/30326993/reproduce-vanilla-packages.sh) | Same crash with **no MicroCeph or snap at all** — plain `.deb` packages, hand-built cluster, real systemd units. Proves this is an upstream Ceph defect, not MicroCeph's packaging. |

The crash itself, condensed (what `reproduce-microceph-minimal.sh` does): enable RGW on a single MicroCeph cluster, then send one signed admin request. `curl`'s built-in AWS SigV4 support (7.75+) is enough — no scripting required:

```bash
UJ=$(microceph.radosgw-admin user info --uid=sync-user)
AK=$(echo "$UJ" | jq -r .keys[0].access_key)
SK=$(echo "$UJ" | jq -r .keys[0].secret_key)

# no crash — plain shard count:
curl --aws-sigv4 "aws:amz:us-east-1:s3" --user "$AK:$SK" \
"http://localhost:80/admin/log?type=data"
# {"num_objects":128}

# CRASHES radosgw:
curl --aws-sigv4 "aws:amz:us-east-1:s3" --user "$AK:$SK" \
"http://localhost:80/admin/log?type=data&id=0&info"
# curl: (52) Empty reply from server
```

Multisite data sync issues exactly this second request to its peer zone on every sync cycle, which is why `reproduce-microceph-multisite.sh` needs no manual trigger at all — it crash-loops on its own once two zones are paired.

## Expected vs actual

- Expected: `data sync` reaches `caught up with source`; objects written on one zone appear on the other.
- Actual: `data sync` stays at `full sync: 128/128 shards` forever. `radosgw` segfaults every ~30s. `sync status` intermittently prints `failed to retrieve sync info: (2200) Unknown error 2200` — a side-effect of the crash (see below), not the real error.

## Root cause

```
*** Caught signal (Segmentation fault) **
in thread thread_name:io_context_pool
ceph version 20.2.1 (6a49aff47758778a5f5951e731d437c317f72fb2) tentacle (stable)
2: (std::rethrow_exception(...)+0x1e)
3: (int rgw::run_coro<...RGWDataChangesLogInfo...>(...)+0x6dd)
4: (RGWOp_DATALog_ShardInfo::execute(optional_yield)+0x2bd)
5: (rgw_process_authenticated(...)+0xab5)
6: (process_request(...)+0x1df5)
10: make_fcontext()
```

## Not a MicroCeph issue

- MicroCeph's own patches (`patches/0001-*`, `patches/0003-*`) touch only the orchestrator and an SMB stub — nothing in RGW.
- Reproduced by [`reproduce-vanilla-packages.sh`](reproduce-vanilla-packages.sh) on **plain `.deb` packages, no snap**: `ceph-mon`/`ceph-osd`/`radosgw` 20.2.1 from `ppa:lmlogiudice/ceph-tentacle-rc` (the same PPA `tentacle/candidate` pulls), a hand-built single-node cluster (real systemd units, no cephadm/container). Same crash, same commit hash. `systemd` logged the exit explicitly: `Main process exited, code=dumped, status=11/SEGV`.
- Squid 19.2.3 (`squid/stable`), identical procedure: no crash, data sync converges normally.

This is a pure upstream Ceph 20.2.1 defect, not something in MicroCeph's snap build, patches, or confinement.

Contributor guide

Open the contributing guide

Research direction

Start with reproduce-vanilla-packages.sh to confirm the crash outside MicroCeph, then inspect the upstream stack frames around RGWOp_DATALog_ShardInfo::execute and RGWDataChangesLogInfo. Done means the /admin/log?type=data&id=0&info request no longer segfaults and multisite data sync reaches caught up with source on Ceph 20.2.1.

Written by the indexing model from the issue text.

Assessment

Domain
backend, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.