apache / apache/beam

[Feature Request]: Support read from / write to AWS Glue catalog tables backed by AWS S3 data

Open
#26,429 4 comments 0 reactions 0 assignees View on GitHub
awaiting triage io java new feature P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What would you like to happen?

## Overview

Glue catalog is a serverless metadata storage (databases, tables, schemas, partitions, connectors, etc). More: https://docs.aws.amazon.com/glue/latest/dg/glue-connections.html

It would be cool to have Beam supporting smth like:

```
GlueIO.write()
.withClientConfiguration(awsClientConfiguration)
.withDatabaseName("my_glue_db")
.withTableName("my_glue_table")
.withIOType(org.apache.beam.sdk.io.FileIO.class)
.withIOConfig(... configs for FileIO / JdbcIO / ... )
.withSchemaUpdateStrategy(ADD_NEW_COLUMNS | DISABLED)
```

## Other existing implementations

1. For AWS S3-backed tables Spark on AWS EMR supports writing to a Glue table in a similar way it does for Hive Metastore tables:

```
df.write.saveAsTable("glue_db.glue_table")
```

2. Trino supports Glue catalog too - https://trino.io/docs/current/connector/hive.html - in a similar fashion Spark does - using it as a replacement for metadata of tables which are stored in some filesystem.

3. AWS Glue job (which is AWS fork of Spark) supports other types of storages: Mongo, RDS, etc

4. Flink seems to have it as work-in-progress: https://github.com/apache/flink-connector-aws/pull/47

## Notes on possible implementation

Beam has `HCatalogIO` implementation - https://beam.apache.org/documentation/io/built-in/hcatalog/ - but it does not seem to be a good place for `GlueIO`:
- it is highly coupled with Hive dependencies
- it can not run on Java 11: https://github.com/apache/beam/issues/21299
- Potentially `GlueIO` can support more storage types besides file systems - adding such into `GlueIO` can be easier
- `HCatalogIO` currently doesn't have any machinery for AWS auth, coders, etc

### Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

### Issue Components

- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [X] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.