apache / apache/polaris

Amazon S3 Tables Integration

Open
#577 10 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
2.1k
Forks
522
Avg merge
1d 22h
Merged PRs (30d)
137

Description

Polaris is designed to act as a **REST facade for S3 tables**, enabling both **read** and **write** operations by interacting with the S3 table API. Polaris registers an S3 table using its **metadata location**. A flag will be needed to label the new Iceberg table is a s3 table. Below is a summary of the proposed approach:
### **Read Path**
- The **LoadTable** endpoint in Polaris will call the S3 table API [`get_table_metadata_location`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_s3TableBuckets_GetTableMetadataLocation.html) to fetch the `metadata.json` file.
- Polaris will then **serialize** the content of the `metadata.json` into a **LoadTableResponse** to return to the client.

### **Write Path**
When the table is updated, Polaris will:
1. Gather the changes and generate a new `metadata.json`.
2. Use the S3 table API [`update-table-metadata-location`](https://docs.aws.amazon.com/cli/latest/reference/s3tables/update-table-metadata-location.html) to commit the new metadata.

### **AuthZ/AuthN**
We need to ensure that the AWS role used for creating the Polaris catalog has the read and write privileges of the s3 table.

### Describe alternatives you've considered

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the LoadTable endpoint and the S3 table APIs get_table_metadata_location and update-table-metadata-location. Trace how metadata.json becomes a LoadTableResponse and how table updates generate new metadata. Done means read and write paths work through S3 Tables and the catalog's AWS role has the required read and write privileges.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
authentication, backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.