apache / apache/datafusion

[datafusion-contrib] AWS Glue Integration

Open
#2,206 2 comments 4 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

This has been discussed in various places, https://github.com/apache/arrow-datafusion/issues/907 and https://github.com/datafusion-contrib/datafusion-objectstore-s3/pull/53 to name a few, so creating an issue for visibility.

**Describe the solution you'd like**

I would propose creating a new datafusion-contrib crate, perhaps `datafusion-catalog-glue`, which communicates with an [AWS Glue Data Catalog](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api.html).

I'll leave the exact design for whoever picks this up, but I might expect something along the following lines.

* Create a `GlueCatalog` with an optional catalog ID
* Provide a `async fn GlueCatalog::list_databases(&self) -> Vec` to list the databases
* Provide a `async fn GlueCatalog::get_database(&self, name: &str) -> Result` to get a database
* Implement `SchemaProvider` for `GlueDatabase`

I think it should be possible to reuse the `FileScanConfig` structure used by `ListingTable` to simplify implementation of the `TableProvider`.

**Describe alternatives you've considered**

We could not support AWS Glue

**Additional context**

This will help with https://github.com/datafusion-contrib/datafusion-objectstore-s3/pull/53 by alleviating the need to infer the schema from the files on every query, and only listing files in non-pruned partitions.

This may need to depend on https://github.com/datafusion-contrib/datafusion-objectstore-s3 as I think it will still need to list S3 in order to get the files within a given partition.

The Glue API is not the snappiest of things, so a future extension might be to cache the metadata returned, as is done by the [Java client](https://github.com/awslabs/aws-glue-data-catalog-client-for-apache-hive-metastore#enabling-client-side-caching-for-catalog).

Contributor guide

Open the contributing guide

Research direction

Start by reading issue 907, datafusion-objectstore-s3 pull request 53, and the AWS Glue API documentation. Then examine DataFusion's SchemaProvider, FileScanConfig, and ListingTable concepts to define the new datafusion-catalog-glue crate. Done means GlueCatalog and GlueDatabase support the proposed database and schema operations and integrate with partition-aware table access.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
cloud, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.