Amazon S3 Tables Integration
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 522
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 137
Description
Polaris is designed to act as a **REST facade for S3 tables**, enabling both **read** and **write** operations by interacting with the S3 table API. Polaris registers an S3 table using its **metadata location**. A flag will be needed to label the new Iceberg table is a s3 table. Below is a summary of the proposed approach:
### **Read Path**
- The **LoadTable** endpoint in Polaris will call the S3 table API [`get_table_metadata_location`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_s3TableBuckets_GetTableMetadataLocation.html) to fetch the `metadata.json` file.
- Polaris will then **serialize** the content of the `metadata.json` into a **LoadTableResponse** to return to the client.
### **Write Path**
When the table is updated, Polaris will:
1. Gather the changes and generate a new `metadata.json`.
2. Use the S3 table API [`update-table-metadata-location`](https://docs.aws.amazon.com/cli/latest/reference/s3tables/update-table-metadata-location.html) to commit the new metadata.
### **AuthZ/AuthN**
We need to ensure that the AWS role used for creating the Polaris catalog has the read and write privileges of the s3 table.
### Describe alternatives you've considered
_No response_
Contributor guide
Research direction
Start with the LoadTable endpoint and the S3 table APIs get_table_metadata_location and update-table-metadata-location. Trace how metadata.json becomes a LoadTableResponse and how table updates generate new metadata. Done means read and write paths work through S3 Tables and the catalog's AWS role has the required read and write privileges.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- authentication, backend, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100