apache / apache/datafusion

[datafusion-contrib] Support Stand Alone Hive MetaStore

Open
#2,209 0 comments 5 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Whilst [AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html) can act as a Hive compatible metastore, and #2206 tracks adding support for it, this is of course not an option for people running outside of AWS. It would be good to provide some story for these users.

**Describe the solution you'd like**

I'm not very familiar with the Hive ecosystem, but it would appear that when people refer to the Hive metastore, they are referring to a client-side implementation, as opposed to a some service like Zookeeper, or Kafka.

I found this exceptionally confusing, but there appear to be three options:

1. An implementation using an embedded Derby database
2. An implementation using a remote JDBC database
3. An implementation using a remote Hive server

I don't think 1. is useful for DataFusion's purposes, and 2. is likely tricky to implement without running into subtle incompatibilities with the Java version, I therefore think 3. is likely the best option. From my understanding this effectively takes the implementation of 2. and runs it as a network service with a [thrift API](https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift). This should be relatively straightforward to support and can likely share a lot of code with any AWS Glue support.

**Describe alternatives you've considered**

We could not support this

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the related AWS Glue issue #2206 and the Hive metastore thrift API linked in this issue. Clarify which remote Hive server behavior DataFusion should support and how it can share the AWS Glue approach. Done means providing a usable standalone Hive metastore option for users outside AWS.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, rust
Domain
backend-api-design, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.