borgbackup / borgbackup/borg

tests: pick a maintained S3 server for the CI S3 tests

Open
#10,361 0 comments 0 reactions 0 assignees View on GitHub
testsuite
Dominant language
Python
Stars
13.7k
Forks
875
Avg merge
11h 15m
Merged PRs (30d)
192

Description

Split off from #10360, which fixed CI but only bought us time.

## What happened

The CI S3 tests run against a MinIO server downloaded at job setup. MinIO has withdrawn its community binaries:

- `https://dl.min.io/server/minio/release/linux-{amd64,arm64}/minio` and the matching `mc` URLs now answer **HTTP 410 Gone**. `curl -f` exits 22, so the step failed before borg was even installed.
- Their GitHub releases from `RELEASE.2025-10-15T17-29-55Z` on carry **0 assets**. The last release that still publishes binaries is `RELEASE.2025-09-07T16-13-09Z`.

That broke every PR and would have broken master on its next run: 3 `native_tests` jobs, both `oldglibc_binary` jobs, the codecov upload (no tests ran, so no coverage), plus fail-fast cancellations.

#10360 pins those last two releases (server + `mc`) from GitHub and verifies each download's sha256. CI is green again, but we are now pinned to a frozen snapshot of a project that has stopped shipping community binaries. It will not get security fixes, and nothing stops the remaining assets from being withdrawn too.

## Where S3 is used in CI

1. `.github/workflows/ci.yml`, step *Install and configure MinIO S3 server (test only)* — starts a server and sets `BORG_TEST_S3_REPO`, which enables `test_s3_repo_basics` in `src/borg/testsuite/archiver/remote_repo_test.py`. That test is one round trip: `repo-create`, `create`, `repo-list`, `list`, `extract`, `delete -a`, `repo-delete`.
2. `.github/workflows/ci.yml`, step *Smoke-test the binary against an S3 server* in `oldglibc_binary` — runs the same shape of check against the frozen PyInstaller binary, to exercise the botocore models the spec bundles. It creates the bucket with boto3 rather than `mc`.

Only the first step needs the `mc` client at all, and only to create a bucket — step 2 already shows that boto3 does the job, so a replacement could drop that download entirely.

## What a replacement has to support

borgstore's S3 backend (`borgstore/backends/s3.py`) is what has to work, not the AWS surface at large:

- `put_object`, `get_object`, `head_object`, `delete_object`, `copy_object`
- `list_objects_v2` with `Prefix`, `Delimiter`, `MaxKeys` and continuation tokens
- **ranged GETs** (`get_object(..., Range=...)`) — these are load-bearing, not an optimization: borg reads individual objects out of packs by range, and `borg compact` rewrites packs through the generic `defrag` in `borgstore/backends/_base.py`, which is ranged `load()` calls plus one `put_object`.
- No multipart upload is used.
- Note the existing compatibility workaround: the backend sets `request_checksum_calculation="when_required"` / `response_checksum_validation="when_required"` and strips `x-amz-checksum-crc32` and `x-amz-sdk-checksum-algorithm` from requests. A candidate server should not depend on those headers.

## Candidates worth evaluating

| | License / runtime | Distribution | Notes |
|---|---|---|---|
| **Garage** | AGPL-3.0, Rust | static binaries, published per release | actively maintained, publishes an S3 compatibility matrix; needs a small config file |
| **SeaweedFS** | Apache-2.0, Go | GitHub release binaries | `weed s3` subcommand; large project, S3 gateway is one component |
| **s3proxy** (gaul/s3proxy) | Apache-2.0, Java | jar | maps S3 onto a filesystem; needs a JRE on the runner |
| **adobe/s3mock** | Apache-2.0, Java | jar / container | a test double rather than a server; fine for a round-trip test, unclear on ranged GETs at our usage |
| **moto** (`moto[s3,server]`) | Apache-2.0, Python | **pip** | no external download at all — the runners already have Python and we already install requirements; it is a mock, so fidelity needs checking |
| MinIO pinned | AGPL-3.0, Go | last GitHub release with assets | status quo after #10360 |

`moto` is attractive for CI specifically because it removes the external binary download that caused this outage in the first place; whether its ranged GET and `list_objects_v2` behaviour is faithful enough for borgstore is the thing to test. Garage looks like the strongest choice if we want a real server.

## Suggested next step

Stand one or two candidates up in a branch, point `BORG_TEST_S3_REPO` at them and run `test_s3_repo_basics` plus a `borg compact` on an S3 repo (compact is the part that leans hardest on ranged reads, and it is not covered by the current S3 test — worth adding either way).

Not urgent, since #10360 unblocked CI, but worth doing before the pinned assets disappear too.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.