[Feature] Support S3-compatible object storage (S3/MinIO) filesystem
- Dominant language
- C++
- Stars
- 131
- Forks
- 49
- PR merge metrics
- No merged PRs in 30d
Description
## Description
Add support for S3-compatible object storage as a filesystem backend in paimon-cpp, including:
- **AWS S3** — native S3 access
- **MinIO** — S3-compatible self-hosted object storage
- Other S3-compatible services (e.g. Alibaba OSS, Ceph RGW, etc.)
## Motivation
Many data lakehouse deployments use object storage (S3, MinIO, OSS) as the underlying storage layer. Currently paimon-cpp appears to support only local filesystem, with HDFS support also in progress (#33). S3/MinIO support would significantly broaden the adoption scenarios for paimon-cpp, especially:
1. Cloud-native deployments on AWS / Alibaba Cloud / other cloud providers
2. On-premise environments using MinIO as the storage backend
3. Hybrid setups where data lives on object storage
## Possible Implementation
- Implement a `S3FileSystem` class conforming to the existing `FileSystem` interface
- Use the AWS C++ SDK (`aws-sdk-cpp`) with S3 module, or a lightweight alternative like `aws-c-s3`
- Support configurable endpoint, region, access key, secret key, and path-style access (for MinIO compatibility)
- The Java version of Paimon already has S3 filesystem support, which can serve as a reference
## Additional Context
Related: #33 (HDFS filesystem support) — once the filesystem abstraction is mature, adding new backends like S3 should be straightforward.
Contributor guide
Assessment
This issue has not been assessed yet.