apache / apache/opendal

Tracking issue of RFC-7948: S3 provider presets

Open
#8,003 0 comments 1 reaction 0 assignees View on GitHub
enhancement releases-note/feat research rust services/s3
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

## Goal

Track provider-specific presets built on the shared S3 runtime proposed in #7948.

This issue uses **preset** instead of **profile** to avoid confusion with AWS credential profiles. A preset owns a provider-specific scheme, a small configuration allowlist, defaults, validation, and service identity. It does not store credentials or expose `S3Config` wholesale. Existing `s3` remains the escape hatch for the complete S3 configuration surface.

Capability changes are out of scope for this tracker.

## Admission criteria

Add a preset only when all of the following are true:

- The provider has an official and maintained S3-compatible API contract.
- OpenDAL can define a useful provider-owned config allowlist instead of copying `S3Config`.
- The provider has stable endpoint or region rules, or a clear reason to require an explicit endpoint.
- Authentication inputs and supported credential sources are documented.
- Invalid or unsupported fields can fail with `ConfigInvalid` instead of being ignored.
- OpenDAL can maintain unit tests and a provider behavior fixture.
- A native OpenDAL service does not already provide the same user experience, or the S3 preset has a documented advantage.

## Initial presets

- [ ] **Cloudflare R2** — `account_id` or `endpoint`, optional jurisdiction, `auto` signing region, direct or temporary credentials. See [R2 S3 documentation](https://developers.cloudflare.com/r2/get-started/s3/).
- [ ] **MinIO** — required endpoint, optional region with `auto` default, direct or temporary credentials, and explicit anonymous mode. See [MinIO S3 SDK](https://github.com/minio/minio-go).

## Managed-service candidates

These providers have useful endpoint or region rules and should be evaluated next:

- [ ] **Tigris** — fixed global endpoint and `auto` signing region. See [Tigris S3 documentation](https://www.tigrisdata.com/docs/sdks/s3/).
- [ ] **Wasabi** — derive `https://s3..wasabisys.com` from a required region. See [Wasabi service URLs](https://docs.wasabi.com/docs/service-urls-for-wasabis-storage-regions).
- [ ] **DigitalOcean Spaces** — derive `.digitaloceanspaces.com` from a required region. See [Spaces API documentation](https://docs.digitalocean.com/reference/api/spaces/).
- [ ] **Scaleway Object Storage** — derive `https://s3..scw.cloud` from a required region. See [Scaleway Object Storage concepts](https://www.scaleway.com/en/docs/object-storage/concepts/).
- [ ] **QingStor Object Storage** — confirm the maintained endpoint, region, authentication, and API contracts before proposing a config allowlist.

## Deployment-specific candidates

These may still benefit from independent config types, but their defaults vary by deployment. Each needs evidence that a preset improves on explicit `s3` configuration:

- [ ] **SeaweedFS** — require an endpoint; evaluate signing-region and anonymous-mode defaults. See the [SeaweedFS S3 quick start](https://github.com/seaweedfs/seaweedfs#quick-start-with-weed-mini).
- [ ] **Ceph RADOS Gateway** — require an endpoint; research region, addressing, and authentication variants before defining an allowlist. See [Ceph Object Gateway documentation](https://docs.ceph.com/en/latest/radosgw/).

## Providers with an existing native service

Do not prioritize S3 presets for the following providers unless a separate issue demonstrates a concrete advantage over the native OpenDAL service:

- Alibaba Cloud OSS (`oss`)
- Tencent Cloud COS (`cos`)
- Google Cloud Storage (`gcs`)
- Backblaze B2 (`b2`)

AWS S3 already uses the `s3` service and is not a preset candidate.

## Per-preset checklist

Before implementing a candidate, create or link a dedicated issue that defines:

- [ ] The public config fields and their types.
- [ ] URI authority and path semantics.
- [ ] Endpoint derivation and default values.
- [ ] Credential lookup and anonymous-access rules.
- [ ] Invalid field combinations and redacted error/debug output.
- [ ] Scheme registration and provider identity.
- [ ] Unit tests for deserialization, derivation, and validation.
- [ ] A behavior-test fixture that constructs the provider through its preset scheme.
- [ ] User-facing documentation and required generated binding updates.

The current manually documented providers are listed in [`compatible_services.md`](https://github.com/apache/opendal/blob/main/core/services/s3/src/compatible_services.md).

Contributor guide

Open the contributing guide

Research direction

Start with the shared S3 runtime proposed in #7948 and the manually documented providers in core/services/s3/src/compatible_services.md. Choose a candidate only after reviewing its linked provider documentation and admission criteria, then create or link a dedicated issue covering config fields, endpoint and credential rules, validation, registration, tests, behavior fixtures, documentation, and generated bindings. Done means those per-preset requirements are defined and the tracker is updated.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.