[Bug report] Iceberg maintenance jobs: --spark-conf does not block overriding reserved Spark configs despite documentation claiming otherwise
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
### Version
main branch
### Describe what's wrong
The `printUsage()` output of `IcebergExpireSnapshotsJob`, `IcebergRewriteDataFilesJob`, and `IcebergUpdateStatsAndMetricsJob` all state:
> Note: Cannot override catalog, extensions, or app name configs
However, the actual code never enforces this restriction. Custom configs passed via `--spark-conf` are applied directly to `SparkSession.Builder` without any filtering:
```java
// IcebergExpireSnapshotsJob.java:136-148
// IcebergRewriteDataFilesJob.java:148-160
// IcebergUpdateStatsAndMetricsJob.java:105-110
Map customConfigs = IcebergJobUtils.parseCustomSparkConfigs(sparkConfJson);
for (Map.Entry entry : customConfigs.entrySet()) {
sparkBuilder.config(entry.getKey(), entry.getValue());
}
### Error message and/or stacktrace
none
### How to reproduce
# This should fail but currently succeeds silently
bin/run-iceberg-expire-snapshots.sh \
--catalog iceberg_prod --table db.sample \
--spark-conf '{"spark.sql.extensions":"org.apache.spark.sql.SparkSessionExtensions"}'
### Additional context
- The reserved keys are already defined in `IcebergSparkConfigUtils.buildTemplateSparkConfigs()`: `spark.master`, `spark.sql.extensions`, `spark.sql.catalog.*`
- `IcebergJobUtils.parseCustomSparkConfigs` is the natural place to add the filter
- Additionally, `--older-than` timestamps and `--table` identifiers are not validated before being interpolated into SQL, which is inconsistent with the existing `validateRetainLast()` / `validateStrategy()` validators
Contributor guide
Research direction
Start with IcebergExpireSnapshotsJob.java, IcebergRewriteDataFilesJob.java, IcebergUpdateStatsAndMetricsJob.java, and IcebergJobUtils.parseCustomSparkConfigs; compare them with IcebergSparkConfigUtils.buildTemplateSparkConfigs(). Reproduce the --spark-conf example, then check the existing validateRetainLast() and validateStrategy() validators. Done means reserved Spark settings and the noted timestamp and table inputs are rejected consistently with the documented behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100