apache / apache/iceberg

Iceberg-spark-runtime-4.1_2.13-1.11.0 fails in Spark-only runtime when Flink classes are absent

Open
#16,856 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 16h
Merged PRs (30d)
129

Description

### Apache Iceberg version

1.11.0 (latest release)

### Query engine

Spark

### Please describe the bug 🐞

Using the official iceberg-spark-runtime-4.1_2.13-1.11.0.jar in a Spark-only runtime fails when Flink table classes are not present.

FormatModelRegistry eagerly tries to register org.apache.iceberg.flink.data.FlinkFormatModels during static initialization. That code references Flink
classes such as org.apache.flink.table.data.RowData.

If those Flink classes are absent, Spark reads fail with:

─ text
java.lang.NoClassDefFoundError: Could not initialize class org.apache.iceberg.formats.FormatModelRegistry
...
Caused by: java.lang.NoClassDefFoundError: org/apache/flink/table/data/RowData
at org.apache.iceberg.flink.data.FlinkFormatModels.register(FlinkFormatModels.java:30)
at org.apache.iceberg.formats.FormatModelRegistry.registerSupportedFormats(FormatModelRegistry.java:207)
at org.apache.iceberg.formats.FormatModelRegistry.(FormatModelRegistry.java:69)

This happens in the normal Spark reader path (BaseBatchReader), even after disabling Auron/native execution.

Steps to reproduce

1. Start Spark 4.1
2. Add iceberg-spark-runtime-4.1_2.13-1.11.0.jar
3. Read any Iceberg table in Spark SQL

Expected behavior

A Spark-only runtime should be able to use the Spark runtime jar without requiring Flink runtime classes.

Notes

This looks similar to #16720, but with missing Flink classes instead of missing ORC classes.

It seems FormatModelRegistry should skip optional engine-specific registrations when transitive dependencies are absent.

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start with FormatModelRegistry.registerSupportedFormats and its static initialization, then inspect FlinkFormatModels.register and the BaseBatchReader path described in the report. Reproduce the failure with Spark 4.1, the iceberg-spark-runtime-4.1_2.13-1.11.0.jar, and a Spark SQL Iceberg read. Done means a Spark-only runtime can read an Iceberg table without Flink classes present.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.