alibaba / alibaba/paimon-cpp

[Feature] Support S3-compatible object storage (S3/MinIO) filesystem

Open
#357 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
131
Forks
49
PR merge metrics
No merged PRs in 30d

Description

## Description

Add support for S3-compatible object storage as a filesystem backend in paimon-cpp, including:

- **AWS S3** — native S3 access
- **MinIO** — S3-compatible self-hosted object storage
- Other S3-compatible services (e.g. Alibaba OSS, Ceph RGW, etc.)

## Motivation

Many data lakehouse deployments use object storage (S3, MinIO, OSS) as the underlying storage layer. Currently paimon-cpp appears to support only local filesystem, with HDFS support also in progress (#33). S3/MinIO support would significantly broaden the adoption scenarios for paimon-cpp, especially:

1. Cloud-native deployments on AWS / Alibaba Cloud / other cloud providers
2. On-premise environments using MinIO as the storage backend
3. Hybrid setups where data lives on object storage

## Possible Implementation

- Implement a `S3FileSystem` class conforming to the existing `FileSystem` interface
- Use the AWS C++ SDK (`aws-sdk-cpp`) with S3 module, or a lightweight alternative like `aws-c-s3`
- Support configurable endpoint, region, access key, secret key, and path-style access (for MinIO compatibility)
- The Java version of Paimon already has S3 filesystem support, which can serve as a reference

## Additional Context

Related: #33 (HDFS filesystem support) — once the filesystem abstraction is mature, adding new backends like S3 should be straightforward.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.