[datafusion-contrib] AWS Glue Integration
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
This has been discussed in various places, https://github.com/apache/arrow-datafusion/issues/907 and https://github.com/datafusion-contrib/datafusion-objectstore-s3/pull/53 to name a few, so creating an issue for visibility.
**Describe the solution you'd like**
I would propose creating a new datafusion-contrib crate, perhaps `datafusion-catalog-glue`, which communicates with an [AWS Glue Data Catalog](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api.html).
I'll leave the exact design for whoever picks this up, but I might expect something along the following lines.
* Create a `GlueCatalog` with an optional catalog ID
* Provide a `async fn GlueCatalog::list_databases(&self) -> Vec` to list the databases
* Provide a `async fn GlueCatalog::get_database(&self, name: &str) -> Result` to get a database
* Implement `SchemaProvider` for `GlueDatabase`
I think it should be possible to reuse the `FileScanConfig` structure used by `ListingTable` to simplify implementation of the `TableProvider`.
**Describe alternatives you've considered**
We could not support AWS Glue
**Additional context**
This will help with https://github.com/datafusion-contrib/datafusion-objectstore-s3/pull/53 by alleviating the need to infer the schema from the files on every query, and only listing files in non-pruned partitions.
This may need to depend on https://github.com/datafusion-contrib/datafusion-objectstore-s3 as I think it will still need to list S3 in order to get the files within a given partition.
The Glue API is not the snappiest of things, so a future extension might be to cache the metadata returned, as is done by the [Java client](https://github.com/awslabs/aws-glue-data-catalog-client-for-apache-hive-metastore#enabling-client-side-caching-for-catalog).
Contributor guide
Research direction
Start by reading issue 907, datafusion-objectstore-s3 pull request 53, and the AWS Glue API documentation. Then examine DataFusion's SchemaProvider, FileScanConfig, and ListingTable concepts to define the new datafusion-catalog-glue crate. Done means GlueCatalog and GlueDatabase support the proposed database and schema operations and integrate with partition-aware table access.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, rust
- Domain
- cloud, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100