[Feature Request]: Support read from / write to AWS Glue catalog tables backed by AWS S3 data
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What would you like to happen?
## Overview
Glue catalog is a serverless metadata storage (databases, tables, schemas, partitions, connectors, etc). More: https://docs.aws.amazon.com/glue/latest/dg/glue-connections.html
It would be cool to have Beam supporting smth like:
```
GlueIO.write()
.withClientConfiguration(awsClientConfiguration)
.withDatabaseName("my_glue_db")
.withTableName("my_glue_table")
.withIOType(org.apache.beam.sdk.io.FileIO.class)
.withIOConfig(... configs for FileIO / JdbcIO / ... )
.withSchemaUpdateStrategy(ADD_NEW_COLUMNS | DISABLED)
```
## Other existing implementations
1. For AWS S3-backed tables Spark on AWS EMR supports writing to a Glue table in a similar way it does for Hive Metastore tables:
```
df.write.saveAsTable("glue_db.glue_table")
```
2. Trino supports Glue catalog too - https://trino.io/docs/current/connector/hive.html - in a similar fashion Spark does - using it as a replacement for metadata of tables which are stored in some filesystem.
3. AWS Glue job (which is AWS fork of Spark) supports other types of storages: Mongo, RDS, etc
4. Flink seems to have it as work-in-progress: https://github.com/apache/flink-connector-aws/pull/47
## Notes on possible implementation
Beam has `HCatalogIO` implementation - https://beam.apache.org/documentation/io/built-in/hcatalog/ - but it does not seem to be a good place for `GlueIO`:
- it is highly coupled with Hive dependencies
- it can not run on Java 11: https://github.com/apache/beam/issues/21299
- Potentially `GlueIO` can support more storage types besides file systems - adding such into `GlueIO` can be easier
- `HCatalogIO` currently doesn't have any machinery for AWS auth, coders, etc
### Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [X] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
Contributor guide
Assessment
This issue has not been assessed yet.