apache / apache/gravitino

[Improvement] Remove the dependency of the Spark Hive Connector on Apache Kyuubi

Open
#7,113 0 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### What would you like to be improved?

I want to use Gravitino as a unified catalog for multiple Hive metastores and multiple Iceberg catalogs.
However, when accessing the Gravitino Hive catalog via the Spark Connector, the `catalogImplementation` is forcibly set to “hive” by the Kyuubi Spark-Hive Connector. (It seems this is because Kyuubi has a different purpose than Gravitino.)

As a result, when using SparkSessionCatalog with “hive”, Gravitino requires an unnecessary `hive-site.xml` file, or if `hive-site.xml` is not present, Spark creates a `metastore_db` (derby file DB) directory due to the default HiveConf code.

It makes an unnecessary `metastore_db` file in the working directory whenever a user connects to the Gravitino Hive catalog via the Spark connector, and the HiveSessionCatalog is used instead.

Is there any plan or workaround to resolve this issue, so that Spark does not require `hive-site.xml` or create unnecessary `metastore_db` files when accessing the Gravitino Hive catalog via the Spark connector?

### How should we improve?

Implement a new Spark Hive Connector independent of Kyuubi.

OR

Implement a SparkSessionCatalog for Gravitino (implement org.apache.spark.sql.connector.catalog.CatalogExtension)

Contributor guide

Open the contributing guide

Research direction

Start by reading the Spark Hive Connector integration and the Kyuubi Spark-Hive Connector dependency, focusing on where catalogImplementation is forced to “hive”. Compare the two proposed directions: an independent connector or a SparkSessionCatalog implementing org.apache.spark.sql.connector.catalog.CatalogExtension. Done means Gravitino can access its Hive catalog without requiring hive-site.xml or creating metastore_db files.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.