Iceberg-spark-runtime-4.1_2.13-1.11.0 fails in Spark-only runtime when Flink classes are absent
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 129
Description
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
Using the official iceberg-spark-runtime-4.1_2.13-1.11.0.jar in a Spark-only runtime fails when Flink table classes are not present.
FormatModelRegistry eagerly tries to register org.apache.iceberg.flink.data.FlinkFormatModels during static initialization. That code references Flink
classes such as org.apache.flink.table.data.RowData.
If those Flink classes are absent, Spark reads fail with:
─ text
java.lang.NoClassDefFoundError: Could not initialize class org.apache.iceberg.formats.FormatModelRegistry
...
Caused by: java.lang.NoClassDefFoundError: org/apache/flink/table/data/RowData
at org.apache.iceberg.flink.data.FlinkFormatModels.register(FlinkFormatModels.java:30)
at org.apache.iceberg.formats.FormatModelRegistry.registerSupportedFormats(FormatModelRegistry.java:207)
at org.apache.iceberg.formats.FormatModelRegistry.(FormatModelRegistry.java:69)
This happens in the normal Spark reader path (BaseBatchReader), even after disabling Auron/native execution.
Steps to reproduce
1. Start Spark 4.1
2. Add iceberg-spark-runtime-4.1_2.13-1.11.0.jar
3. Read any Iceberg table in Spark SQL
Expected behavior
A Spark-only runtime should be able to use the Spark runtime jar without requiring Flink runtime classes.
Notes
This looks similar to #16720, but with missing Flink classes instead of missing ORC classes.
It seems FormatModelRegistry should skip optional engine-specific registrations when transitive dependencies are absent.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start with FormatModelRegistry.registerSupportedFormats and its static initialization, then inspect FlinkFormatModels.register and the BaseBatchReader path described in the report. Reproduce the failure with Spark 4.1, the iceberg-spark-runtime-4.1_2.13-1.11.0.jar, and a Spark SQL Iceberg read. Done means a Spark-only runtime can read an Iceberg table without Flink classes present.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 64/100