block / block/buzz

Git object-store A3 conformance probe fails on GCS S3-interop (`if_match_race`) — consider a native GCS backend

Open
#2,470 1 comment 2 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

## Environment

- Relay: `ghcr.io/block/buzz:sha-2d26db6` (`2d26db6d8a7381ff0dc9b50194fd28d42957de74`); reproduced identically on `sha-acfbb1b` (2026-07-22 main)
- Object storage: **Google Cloud Storage via the S3-interop XML API** (`BUZZ_S3_ENDPOINT=https://storage.googleapis.com`, HMAC service-account credentials)
- Deployment: GKE, chart `buzz` 0.1.6, external Postgres/Redis

## What happens

Relay startup is fatal at the A3 git conformance gate:

```
INFO running git object-store conformance probe (A3 gate)
Error: git conformance probe failed: conformance probe failed in phase 'if_match_race' (round 0, key probe/pointer-0d592977-…)
```

Media paths on the same bucket/credentials work (`Media storage connected`;
put/get verified end-to-end through the app).

## Why

The probe is right to fail: GCS's S3-interoperability layer does not honor
S3-style conditional writes (`If-Match`/`If-None-Match` preconditions on PUT),
so the linearizable pointer CAS that `run_conformance_probe` verifies (and
that the manifest-pointer protocol needs) cannot be satisfied. This makes
git-on-object-storage unusable on GCS for any deployer using the documented
S3 configuration — the probe converts that into a hard startup failure.

## What we did (and why it's not a fix)

Booted the pilot with `BUZZ_GIT_CONFORMANCE_PROBE=false` and quarantined git
features operationally. That's an availability workaround, not a solution —
the probe's verdict about pointer CAS still stands.

## Proposal

GCS *does* support real CAS — via its native preconditions
(`x-goog-if-generation-match: 0` for create-if-absent, generation-match for
compare-and-swap). Options in rough order of preference:

1. **Native GCS storage backend** for the git object store (JSON/XML API with
generation preconditions) behind the existing storage trait.
2. **Conditional-write shim for GCS endpoints**: detect
`storage.googleapis.com` and translate the pointer-CAS operations to
`x-goog-*` preconditions while leaving plain put/get on the S3 path.
3. At minimum, **document** that git-on-object-storage requires a backend with
S3 conditional-write semantics (AWS S3 ≥ 2024 If-Match support, MinIO, R2?)
and name GCS S3-interop as unsupported — so deployers hit a doc instead of
a fatal probe.

We (Apiary — hosted Buzz on GCP) are happy to test any of these against real
GCS, and would rather contribute the backend than run a MinIO shim in front
of GCS.

Contributor guide

Open the contributing guide

Research direction

Start with the existing storage trait and the run_conformance_probe entry point, then review how the if_match_race phase performs pointer CAS. Confirm the selected GCS approach against the native generation preconditions and the documented S3 configuration. Done means GCS git object storage passes the conformance gate, or its unsupported status is clearly documented without requiring a fatal startup failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, rust
Domain
backend, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.