Ceph RGW rejects quoted ETags in If-Match (412) — relay CAS / S3 conformance probe fails on Ceph-backed object storage
- Dominant language
- Rust
- Stars
- 32.7k
- Forks
- 4.3k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 253
Description
### Summary
When `buzz-relay` is backed by an S3-compatible object store implemented by **Ceph RGW**, every conditional-write (CAS) — including the **startup S3 conformance probe** — fails with HTTP 412, causing the relay to CrashLoop on startup.
### Root cause
Ceph RGW returns ETags in quoted form (e.g. `"deadbeef"`, per RFC 7232) in GET responses, but **rejects quoted ETags in the `If-Match` precondition header** (returns 412) while accepting the **unquoted** form (e.g. `deadbeef`). This is non-compliant with RFC 7232, which requires `If-Match` to carry a quoted entity-tag.
`buzz-relay` takes the ETag it received from a GET response and forwards it **verbatim** into the `If-Match` header of the subsequent PUT. Because the stored ETag is quoted, every `If-Match` carries a quoted value that Ceph RGW rejects → 412 → CAS fails.
Relevant code: [`crates/buzz-relay/src/api/git/store.rs`](https://github.com/block/buzz/blob/main/crates/buzz-relay/src/api/git/store.rs), `put_pointer()`, the `Precond::IfMatch(ETag(tag))` arm:
```rust
Precond::IfMatch(ETag(tag)) => {
headers.insert(
axum::http::header::IF_MATCH,
tag.parse().map_err(|_| {
StoreError::Backend(S3Error::HttpFailWithBody(
400,
format!("invalid etag {tag}"),
))
})?,
);
}
```
This single path covers both the conformance probe (`etag_consistency` phase, which calls `get_pointer` → `put_pointer(IfMatch)`) and `cas_publish.rs` (`Precond::IfMatch(e.clone())`).
### Reproduction
- S3 backend: Ceph RGW ( Reef / any recent release).
- Configure buzz-relay against a Ceph-backed bucket (`BUZZ_S3_ENDPOINT`, `BUZZ_S3_BUCKET`, credentials).
- Start the relay: the startup S3 conformance probe fails with 412; the relay exits / CrashLoops.
Empirically confirmed with a stdlib SigV4 probe against our Ceph RGW:
- GET returns `ETag: "deadbeef"` (quoted).
- `If-Match: "deadbeef"` (quoted) → **412 Precondition Failed**.
- `If-Match: deadbeef` (unquoted) → **200 OK**.
### Suggested fix
Normalize the ETag before inserting it into `If-Match` — strip surrounding double quotes:
```rust
Precond::IfMatch(ETag(tag)) => {
// Ceph RGW rejects quoted If-Match values (RFC 7232 non-compliant) but
// accepts the unquoted form. AWS S3 and MinIO accept both, so stripping
// is safe across backends.
let unquoted = tag.trim_matches('"');
headers.insert(
axum::http::header::IF_MATCH,
unquoted.parse().map_err(|_| {
StoreError::Backend(S3Error::HttpFailWithBody(
400,
format!("invalid etag {unquoted}"),
))
})?,
);
}
```
This is harmless for AWS S3 and MinIO, which accept both the quoted and unquoted forms, so it does not regress those backends.
Alternatively, a more general fix could normalize the ETag on read (when received from GET) so callers always see the unquoted form, but that is a larger change with broader blast radius.
### Environment
- buzz-relay built from chart tag `chart-v0.1.6`
- S3 backend: Ceph RGW
Happy to open a PR if this approach is acceptable.
Contributor guide
Assessment
This issue has not been assessed yet.