tests: pick a maintained S3 server for the CI S3 tests
- Dominant language
- Python
- Stars
- 13.7k
- Forks
- 875
- Avg merge
- 11h 15m
- Merged PRs (30d)
- 192
Description
Split off from #10360, which fixed CI but only bought us time.
## What happened
The CI S3 tests run against a MinIO server downloaded at job setup. MinIO has withdrawn its community binaries:
- `https://dl.min.io/server/minio/release/linux-{amd64,arm64}/minio` and the matching `mc` URLs now answer **HTTP 410 Gone**. `curl -f` exits 22, so the step failed before borg was even installed.
- Their GitHub releases from `RELEASE.2025-10-15T17-29-55Z` on carry **0 assets**. The last release that still publishes binaries is `RELEASE.2025-09-07T16-13-09Z`.
That broke every PR and would have broken master on its next run: 3 `native_tests` jobs, both `oldglibc_binary` jobs, the codecov upload (no tests ran, so no coverage), plus fail-fast cancellations.
#10360 pins those last two releases (server + `mc`) from GitHub and verifies each download's sha256. CI is green again, but we are now pinned to a frozen snapshot of a project that has stopped shipping community binaries. It will not get security fixes, and nothing stops the remaining assets from being withdrawn too.
## Where S3 is used in CI
1. `.github/workflows/ci.yml`, step *Install and configure MinIO S3 server (test only)* — starts a server and sets `BORG_TEST_S3_REPO`, which enables `test_s3_repo_basics` in `src/borg/testsuite/archiver/remote_repo_test.py`. That test is one round trip: `repo-create`, `create`, `repo-list`, `list`, `extract`, `delete -a`, `repo-delete`.
2. `.github/workflows/ci.yml`, step *Smoke-test the binary against an S3 server* in `oldglibc_binary` — runs the same shape of check against the frozen PyInstaller binary, to exercise the botocore models the spec bundles. It creates the bucket with boto3 rather than `mc`.
Only the first step needs the `mc` client at all, and only to create a bucket — step 2 already shows that boto3 does the job, so a replacement could drop that download entirely.
## What a replacement has to support
borgstore's S3 backend (`borgstore/backends/s3.py`) is what has to work, not the AWS surface at large:
- `put_object`, `get_object`, `head_object`, `delete_object`, `copy_object`
- `list_objects_v2` with `Prefix`, `Delimiter`, `MaxKeys` and continuation tokens
- **ranged GETs** (`get_object(..., Range=...)`) — these are load-bearing, not an optimization: borg reads individual objects out of packs by range, and `borg compact` rewrites packs through the generic `defrag` in `borgstore/backends/_base.py`, which is ranged `load()` calls plus one `put_object`.
- No multipart upload is used.
- Note the existing compatibility workaround: the backend sets `request_checksum_calculation="when_required"` / `response_checksum_validation="when_required"` and strips `x-amz-checksum-crc32` and `x-amz-sdk-checksum-algorithm` from requests. A candidate server should not depend on those headers.
## Candidates worth evaluating
| | License / runtime | Distribution | Notes |
|---|---|---|---|
| **Garage** | AGPL-3.0, Rust | static binaries, published per release | actively maintained, publishes an S3 compatibility matrix; needs a small config file |
| **SeaweedFS** | Apache-2.0, Go | GitHub release binaries | `weed s3` subcommand; large project, S3 gateway is one component |
| **s3proxy** (gaul/s3proxy) | Apache-2.0, Java | jar | maps S3 onto a filesystem; needs a JRE on the runner |
| **adobe/s3mock** | Apache-2.0, Java | jar / container | a test double rather than a server; fine for a round-trip test, unclear on ranged GETs at our usage |
| **moto** (`moto[s3,server]`) | Apache-2.0, Python | **pip** | no external download at all — the runners already have Python and we already install requirements; it is a mock, so fidelity needs checking |
| MinIO pinned | AGPL-3.0, Go | last GitHub release with assets | status quo after #10360 |
`moto` is attractive for CI specifically because it removes the external binary download that caused this outage in the first place; whether its ranged GET and `list_objects_v2` behaviour is faithful enough for borgstore is the thing to test. Garage looks like the strongest choice if we want a real server.
## Suggested next step
Stand one or two candidates up in a branch, point `BORG_TEST_S3_REPO` at them and run `test_s3_repo_basics` plus a `borg compact` on an S3 repo (compact is the part that leans hardest on ranged reads, and it is not covered by the current S3 test — worth adding either way).
Not urgent, since #10360 unblocked CI, but worth doing before the pinned assets disappear too.
Contributor guide
Assessment
This issue has not been assessed yet.