apache / apache/datafusion-comet
Codify Iceberg version compatibility
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
With the recent addition of Iceberg 1.11 compatibility to Comet (https://github.com/apache/datafusion-comet/pull/4840) we should codify what versions we will run in CI. #4840 has a table of testing after that PR, which I duplicate below:
CI coverage across Spark and Iceberg versions (rows are the Spark version; columns are the Iceberg runtime paired with it):
| Spark \ Iceberg | 1.5.2 | 1.8.1 | 1.9.1 | 1.10.0 | 1.11.0 |
|-----------------|-------|-------|-------|--------|--------|
| **3.4** | Comet suites — every PR | Iceberg tests — main + `run-iceberg-tests` | — | — | — |
| **3.5** | — | Comet suites — every PR | Iceberg tests — main + `run-iceberg-tests` | Iceberg tests — main + `run-iceberg-tests` | — |
| **4.0** | — | — | — | Comet suites — every PR | — |
| **4.1** | — | — | — | — | Comet suites + Iceberg tests — every PR |
| **4.2** | — | — | — | Comet suites — every PR | — |
**Comet suites** = `CometIcebergNativeSuite` (and the other Iceberg-touching Comet unit suites) in the `pr_build_linux` "scans" bucket; the Iceberg runtime is pulled by the active `spark-*` profile in `spark/pom.xml`. Runs on every PR across all five Spark profiles.
**Iceberg tests** = Apache Iceberg's own `iceberg-spark` / `-extensions` / `-runtime` suites run with Comet enabled, via the `iceberg_1_8` / `iceberg_1_9` / `iceberg_1_10` / `iceberg_1_11` jobs. 1.11 (Spark 4.1) runs on every PR; the older three run on push-to-main or on a PR labeled `run-iceberg-tests`.
We maintain diffs for Iceberg 1.8, 1.9, 1.10, and 1.11 so we can run their Iceberg Java test suites. However, it really seems like each subsequent version is a superset of the former.
We should 1) look at what Iceberg's stated support window is, 2) compare to Spark's stated support window, 3) reconcile with Comet's support/compatibility goals in order to come up with a consistent plan.
Contributor guide
Assessment
This issue has not been assessed yet.