[FEATURE] Add a job-level switch to disable Gravitino Spark connector initialization
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 339
Description
### Describe the feature
Add a job-level configuration to allow Spark applications to skip Gravitino Spark connector initialization while keeping the global `spark.plugins` configuration unchanged.
Proposed configuration:
```properties
spark.sql.gravitino.enabled=false
```
Default value:
```properties
spark.sql.gravitino.enabled=true
```
When `spark.sql.gravitino.enabled=false`, `GravitinoSparkPlugin` can still be loaded by Spark, but the Gravitino driver plugin should skip all Gravitino initialization logic.
### Motivation
Some platforms inject Gravitino Spark connector configurations globally, for example:
```properties
spark.plugins=org.apache.gravitino.spark.connector.plugin.GravitinoSparkPlugin
spark.sql.gravitino.uri=...
spark.sql.gravitino.metalake=...
```
However, not all Spark workloads are compatible with Gravitino today. For example, TiSpark jobs may fail during startup or execution when the Gravitino Spark connector is initialized.
In these cases, users need a per-job way to opt out of Gravitino behavior without requiring the platform to remove the global `spark.plugins` configuration. This would make it easier to adopt Gravitino incrementally while still supporting incompatible or not-yet-supported Spark workloads.
### Describe the solution
Introduce a new Spark configuration:
```java
public static final String GRAVITINO_ENABLED =
GRAVITINO_PREFIX + "enabled";
```
The default behavior should remain unchanged:
```properties
spark.sql.gravitino.enabled=true
```
At the very beginning of `GravitinoDriverPlugin.init()`, before reading or validating `spark.sql.gravitino.uri` and `spark.sql.gravitino.metalake`, check this flag:
```java
if (!conf.getBoolean(GravitinoSparkConfig.GRAVITINO_ENABLED, true)) {
LOG.info("Gravitino Spark connector is disabled.");
return Collections.emptyMap();
}
```
When disabled, the connector should:
- Not validate `spark.sql.gravitino.uri`
- Not validate `spark.sql.gravitino.metalake`
- Not create a `GravitinoClient`
- Not access the Gravitino server
- Not call `loadRelationalCatalogs()`
- Not register Gravitino catalogs into Spark
- Not inject Gravitino SQL extensions
Spark should continue running with native Spark, TiSpark, or user explicitly configured catalogs.
### Additional context
This switch is not intended to:
- Remove `spark.plugins` itself
- Avoid class loading of the Gravitino connector jar
- Disable Gravitino or hybrid extensions manually configured in `spark.sql.extensions`
- Disable Gravitino catalog classes manually configured in `spark.sql.catalog.xxx`
Those configurations should still be controlled by the job submission side.
Suggested tests:
- Default behavior remains unchanged when `spark.sql.gravitino.enabled` is not set
- `spark.sql.gravitino.enabled=false` does not require `spark.sql.gravitino.uri`
- `spark.sql.gravitino.enabled=false` does not require `spark.sql.gravitino.metalake`
- Disabled mode does not create a Gravitino client
- Disabled mode does not register catalogs
- Disabled mode does not inject Gravitino SQL extensions
Contributor guide
Research direction
Start by reading GravitinoDriverPlugin.init() and GravitinoSparkConfig, focusing on how configuration is read before initialization. Add coverage for the default-enabled behavior and the disabled flag, including skipped validation, client creation, catalog registration, and SQL extension injection. Done means disabled jobs run without URI or metalake settings while existing defaults remain unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100