apache / apache/gravitino

[Bug report] Iceberg maintenance jobs: --spark-conf does not block overriding reserved Spark configs despite documentation claiming otherwise

Open
#13,113 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### Version

main branch

### Describe what's wrong

The `printUsage()` output of `IcebergExpireSnapshotsJob`, `IcebergRewriteDataFilesJob`, and `IcebergUpdateStatsAndMetricsJob` all state:

> Note: Cannot override catalog, extensions, or app name configs

However, the actual code never enforces this restriction. Custom configs passed via `--spark-conf` are applied directly to `SparkSession.Builder` without any filtering:

```java
// IcebergExpireSnapshotsJob.java:136-148
// IcebergRewriteDataFilesJob.java:148-160
// IcebergUpdateStatsAndMetricsJob.java:105-110
Map customConfigs = IcebergJobUtils.parseCustomSparkConfigs(sparkConfJson);
for (Map.Entry entry : customConfigs.entrySet()) {
sparkBuilder.config(entry.getKey(), entry.getValue());
}

### Error message and/or stacktrace

none

### How to reproduce

# This should fail but currently succeeds silently
bin/run-iceberg-expire-snapshots.sh \
--catalog iceberg_prod --table db.sample \
--spark-conf '{"spark.sql.extensions":"org.apache.spark.sql.SparkSessionExtensions"}'

### Additional context

- The reserved keys are already defined in `IcebergSparkConfigUtils.buildTemplateSparkConfigs()`: `spark.master`, `spark.sql.extensions`, `spark.sql.catalog.*`
- `IcebergJobUtils.parseCustomSparkConfigs` is the natural place to add the filter
- Additionally, `--older-than` timestamps and `--table` identifiers are not validated before being interpolated into SQL, which is inconsistent with the existing `validateRetainLast()` / `validateStrategy()` validators

Contributor guide

Open the contributing guide

Research direction

Start with IcebergExpireSnapshotsJob.java, IcebergRewriteDataFilesJob.java, IcebergUpdateStatsAndMetricsJob.java, and IcebergJobUtils.parseCustomSparkConfigs; compare them with IcebergSparkConfigUtils.buildTemplateSparkConfigs(). Reproduce the --spark-conf example, then check the existing validateRetainLast() and validateStrategy() validators. Done means reserved Spark settings and the noted timestamp and table inputs are rejected consistently with the documented behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
backend, data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.