apache / apache/gravitino

[Improvement] Spark connector should automatically set spark.sql.catalogImplementation=hive for hive provider catalogs

Open
#9,866 3 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 17h
Merged PRs (30d)
339

Description

### What would you like to be improved?

The Gravitino Spark connector requires users to manually set `spark.sql.catalogImplementation=hive` when using the `hive` catalog provider. This is a leaky abstraction - users shouldn't need to know about underlying Spark/Hive implementation details when Gravitino is meant to be the unified federation layer.

**Current behavior:**

The `hive` provider catalog appears in `SHOW CATALOGS`, but querying it fails with:

```
Caused by: java.lang.AssertionError: assertion failed: Require setting spark.sql.catalogImplementation to `hive` to enable hive support.
at org.apache.kyuubi.spark.connector.hive.HiveTableCatalog.initialize(HiveTableCatalog.scala:124)
at org.apache.gravitino.spark.connector.hive.GravitinoHiveCatalog.createAndInitSparkCatalog(GravitinoHiveCatalog.java:42)
```

**Expected behavior:**

The `GravitinoSparkPlugin` should automatically set `spark.sql.catalogImplementation=hive` when it detects a catalog using the `hive` provider - just like `lakehouse-iceberg` works without requiring any additional Spark-level configuration.

Users should only need:
```python
spark = SparkSession.builder \
.config("spark.plugins", "org.apache.gravitino.spark.connector.plugin.GravitinoSparkPlugin") \
.config("spark.sql.gravitino.uri", "http://localhost:8090") \
.config("spark.sql.gravitino.metalake", "my_metalake") \
.getOrCreate()
```

### Environment
- Gravitino version: 1.1.0
- Spark version: 3.5.3

Contributor guide

Open the contributing guide

Research direction

Start at GravitinoSparkPlugin and trace how it handles catalogs, then read GravitinoHiveCatalog.createAndInitSparkCatalog and HiveTableCatalog.initialize to understand where the missing Spark setting is required. Verify the hive provider works without manual configuration while existing providers such as lakehouse-iceberg continue to work, using the Spark 3.5.3 setup described in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.