apache / apache/datafusion-comet

Codify Iceberg version compatibility

Open
#4,871 3 comments 2 reactions 1 assignee Claimed by @mbutrovich View on GitHub
area:ci area:Iceberg enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

With the recent addition of Iceberg 1.11 compatibility to Comet (https://github.com/apache/datafusion-comet/pull/4840) we should codify what versions we will run in CI. #4840 has a table of testing after that PR, which I duplicate below:

CI coverage across Spark and Iceberg versions (rows are the Spark version; columns are the Iceberg runtime paired with it):

| Spark \ Iceberg | 1.5.2 | 1.8.1 | 1.9.1 | 1.10.0 | 1.11.0 |
|-----------------|-------|-------|-------|--------|--------|
| **3.4** | Comet suites — every PR | Iceberg tests — main + `run-iceberg-tests` | — | — | — |
| **3.5** | — | Comet suites — every PR | Iceberg tests — main + `run-iceberg-tests` | Iceberg tests — main + `run-iceberg-tests` | — |
| **4.0** | — | — | — | Comet suites — every PR | — |
| **4.1** | — | — | — | — | Comet suites + Iceberg tests — every PR |
| **4.2** | — | — | — | Comet suites — every PR | — |

**Comet suites** = `CometIcebergNativeSuite` (and the other Iceberg-touching Comet unit suites) in the `pr_build_linux` "scans" bucket; the Iceberg runtime is pulled by the active `spark-*` profile in `spark/pom.xml`. Runs on every PR across all five Spark profiles.

**Iceberg tests** = Apache Iceberg's own `iceberg-spark` / `-extensions` / `-runtime` suites run with Comet enabled, via the `iceberg_1_8` / `iceberg_1_9` / `iceberg_1_10` / `iceberg_1_11` jobs. 1.11 (Spark 4.1) runs on every PR; the older three run on push-to-main or on a PR labeled `run-iceberg-tests`.

We maintain diffs for Iceberg 1.8, 1.9, 1.10, and 1.11 so we can run their Iceberg Java test suites. However, it really seems like each subsequent version is a superset of the former.

We should 1) look at what Iceberg's stated support window is, 2) compare to Spark's stated support window, 3) reconcile with Comet's support/compatibility goals in order to come up with a consistent plan.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.