[Bug report] Built-in Iceberg jobs soft-fail when Iceberg Spark runtime is missing
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
### Version
main branch
### Describe what's wrong
Built-in Iceberg maintenance jobs (`builtin-iceberg-update-stats`, `builtin-iceberg-rewrite-data-files`, `builtin-iceberg-expire-snapshots`) have several related problems:
1. **Missing Iceberg Spark runtime soft-fails**
Templates configure `IcebergSparkSessionExtensions` / `SparkCatalog`, but do not ship `iceberg-spark-runtime` and leave `jars` empty. Spark only warns when the extension class is missing, so the job continues without Iceberg support. Docs also do not clearly state that operators must supply a matching runtime (for example via `spark.jars`).
2. **JSON parse errors use the wrong CLI flag name**
`IcebergUpdateStatsAndMetricsJob` reports parse failures as `--json-options`, but templates and usage expose `--updater-options` / `--spark-conf`. Rewrite options errors similarly do not name `--options`.
3. **Empty / unresolved optional placeholders become dangling flags**
Optional template args are always listed as `--flag` + `{{placeholder}}`. When `jobConf` omits the key or supplies an empty value, the runtime command keeps dangling flags such as `--updater-options --spark-conf`.
### Error message and/or stacktrace
Missing runtime (Spark only warns today):
```text
WARN SparkSession: Cannot use org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions ...
java.lang.ClassNotFoundException: org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
```
Wrong flag name in parse errors:
```text
Option --json-options is not valid JSON: ...
```
(while the real flag is `--updater-options` or `--spark-conf`)
Dangling empty optional args (from generated spark-submit / process argv):
```text
... --updater-options --spark-conf {...}
```
### How to reproduce
1. **Soft-fail without Iceberg runtime**
Run any built-in Iceberg job against a stock Spark install without `iceberg-spark-runtime` on the classpath. Observe the WARN and that the job continues.
2. **Wrong flag name**
Submit update-stats with invalid JSON for `updater_options` or `spark_conf`. The error mentions `--json-options`.
3. **Dangling empty placeholders**
Submit a built-in Iceberg job while omitting optional `jobConf` keys such as `updater_options`, `options`, `older_than`, or `spark_conf` (or set them to empty). Inspect the generated process arguments and see empty / unresolved optional flags still present.
### Additional context
Prefer documenting that operators must supply a version-matched `iceberg-spark-runtime` rather than hardcoding jars into templates. Jobs should fail fast when Iceberg Spark classes are missing; optional empty placeholders should be omitted from the command; parse errors should name the real CLI flags.
Contributor guide
Assessment
This issue has not been assessed yet.