apache / apache/gravitino

[FEATURE] Add a job-level switch to disable Gravitino Spark connector initialization

Open
#13,117 0 comments 0 reactions 0 assignees View on GitHub
feature
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 17h
Merged PRs (30d)
339

Description

### Describe the feature

Add a job-level configuration to allow Spark applications to skip Gravitino Spark connector initialization while keeping the global `spark.plugins` configuration unchanged.

Proposed configuration:

```properties
spark.sql.gravitino.enabled=false
```

Default value:

```properties
spark.sql.gravitino.enabled=true
```

When `spark.sql.gravitino.enabled=false`, `GravitinoSparkPlugin` can still be loaded by Spark, but the Gravitino driver plugin should skip all Gravitino initialization logic.

### Motivation

Some platforms inject Gravitino Spark connector configurations globally, for example:

```properties
spark.plugins=org.apache.gravitino.spark.connector.plugin.GravitinoSparkPlugin
spark.sql.gravitino.uri=...
spark.sql.gravitino.metalake=...
```

However, not all Spark workloads are compatible with Gravitino today. For example, TiSpark jobs may fail during startup or execution when the Gravitino Spark connector is initialized.

In these cases, users need a per-job way to opt out of Gravitino behavior without requiring the platform to remove the global `spark.plugins` configuration. This would make it easier to adopt Gravitino incrementally while still supporting incompatible or not-yet-supported Spark workloads.

### Describe the solution

Introduce a new Spark configuration:

```java
public static final String GRAVITINO_ENABLED =
GRAVITINO_PREFIX + "enabled";
```

The default behavior should remain unchanged:

```properties
spark.sql.gravitino.enabled=true
```

At the very beginning of `GravitinoDriverPlugin.init()`, before reading or validating `spark.sql.gravitino.uri` and `spark.sql.gravitino.metalake`, check this flag:

```java
if (!conf.getBoolean(GravitinoSparkConfig.GRAVITINO_ENABLED, true)) {
LOG.info("Gravitino Spark connector is disabled.");
return Collections.emptyMap();
}
```

When disabled, the connector should:

- Not validate `spark.sql.gravitino.uri`
- Not validate `spark.sql.gravitino.metalake`
- Not create a `GravitinoClient`
- Not access the Gravitino server
- Not call `loadRelationalCatalogs()`
- Not register Gravitino catalogs into Spark
- Not inject Gravitino SQL extensions

Spark should continue running with native Spark, TiSpark, or user explicitly configured catalogs.

### Additional context

This switch is not intended to:

- Remove `spark.plugins` itself
- Avoid class loading of the Gravitino connector jar
- Disable Gravitino or hybrid extensions manually configured in `spark.sql.extensions`
- Disable Gravitino catalog classes manually configured in `spark.sql.catalog.xxx`

Those configurations should still be controlled by the job submission side.

Suggested tests:

- Default behavior remains unchanged when `spark.sql.gravitino.enabled` is not set
- `spark.sql.gravitino.enabled=false` does not require `spark.sql.gravitino.uri`
- `spark.sql.gravitino.enabled=false` does not require `spark.sql.gravitino.metalake`
- Disabled mode does not create a Gravitino client
- Disabled mode does not register catalogs
- Disabled mode does not inject Gravitino SQL extensions

Contributor guide

Open the contributing guide

Research direction

Start by reading GravitinoDriverPlugin.init() and GravitinoSparkConfig, focusing on how configuration is read before initialization. Add coverage for the default-enabled behavior and the disabled flag, including skipped validation, client creation, catalog registration, and SQL extension injection. Done means disabled jobs run without URI or metalake settings while existing defaults remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.