[Improvement] Remove the dependency of the Spark Hive Connector on Apache Kyuubi
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
### What would you like to be improved?
I want to use Gravitino as a unified catalog for multiple Hive metastores and multiple Iceberg catalogs.
However, when accessing the Gravitino Hive catalog via the Spark Connector, the `catalogImplementation` is forcibly set to “hive” by the Kyuubi Spark-Hive Connector. (It seems this is because Kyuubi has a different purpose than Gravitino.)
As a result, when using SparkSessionCatalog with “hive”, Gravitino requires an unnecessary `hive-site.xml` file, or if `hive-site.xml` is not present, Spark creates a `metastore_db` (derby file DB) directory due to the default HiveConf code.
It makes an unnecessary `metastore_db` file in the working directory whenever a user connects to the Gravitino Hive catalog via the Spark connector, and the HiveSessionCatalog is used instead.
Is there any plan or workaround to resolve this issue, so that Spark does not require `hive-site.xml` or create unnecessary `metastore_db` files when accessing the Gravitino Hive catalog via the Spark connector?
### How should we improve?
Implement a new Spark Hive Connector independent of Kyuubi.
OR
Implement a SparkSessionCatalog for Gravitino (implement org.apache.spark.sql.connector.catalog.CatalogExtension)
Contributor guide
Research direction
Start by reading the Spark Hive Connector integration and the Kyuubi Spark-Hive Connector dependency, focusing on where catalogImplementation is forced to “hive”. Compare the two proposed directions: an independent connector or a SparkSessionCatalog implementing org.apache.spark.sql.connector.catalog.CatalogExtension. Done means Gravitino can access its Hive catalog without requiring hive-site.xml or creating metastore_db files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100