[datafusion-contrib] Support Stand Alone Hive MetaStore
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Whilst [AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html) can act as a Hive compatible metastore, and #2206 tracks adding support for it, this is of course not an option for people running outside of AWS. It would be good to provide some story for these users.
**Describe the solution you'd like**
I'm not very familiar with the Hive ecosystem, but it would appear that when people refer to the Hive metastore, they are referring to a client-side implementation, as opposed to a some service like Zookeeper, or Kafka.
I found this exceptionally confusing, but there appear to be three options:
1. An implementation using an embedded Derby database
2. An implementation using a remote JDBC database
3. An implementation using a remote Hive server
I don't think 1. is useful for DataFusion's purposes, and 2. is likely tricky to implement without running into subtle incompatibilities with the Java version, I therefore think 3. is likely the best option. From my understanding this effectively takes the implementation of 2. and runs it as a network service with a [thrift API](https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift). This should be relatively straightforward to support and can likely share a lot of code with any AWS Glue support.
**Describe alternatives you've considered**
We could not support this
Contributor guide
Research direction
Start by reviewing the related AWS Glue issue #2206 and the Hive metastore thrift API linked in this issue. Clarify which remote Hive server behavior DataFusion should support and how it can share the AWS Glue approach. Done means providing a usable standalone Hive metastore option for users outside AWS.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java, rust
- Domain
- backend-api-design, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100