Upgrade to Spark 4.0: move the default build to Scala 2.13 and Java 17

## Background

Discussed on the dev list in [DISCUSS] Scope and timeline for Apache XTable (incubating) 0.5.0 and future releases. The list agreed to keep this **out of 0.5.0** and make it a headline item for **0.6.0**, with the skipped Spark 3.5 step folded in. Two reasons were given: 0.5.0 already carries six items against a 30 September 2026 cutoff, and the Delta 2.4 to 4.0 jump lands directly on top of "Delta Kernel as the default Delta path" (#886), which would make it hard to attribute a regression to either change.

Filing this to track the work.

## Where we are today

`main` is still on the Spark 3.4 / Scala 2.12 / Java 8 combination:

| Property | Value | Location |
|---|---|---|
| `spark.version` | 3.4.2 | `pom.xml:88` |
| `spark.version.prefix` | 3.4 | `pom.xml:89` |
| `scala.version` | `${scala12.version}` = 2.12.20 | `pom.xml:86` |
| `scala.binary.version` | 2.12 | `pom.xml:87` |
| `javaVersion` | 8, fed to `` | `pom.xml:64`, `pom.xml:712` |
| `delta.version` | 2.4.0 | `pom.xml:92` |
| `iceberg.version` | 1.9.2 | `pom.xml:90` |
| `hudi.version` | 1.2.0 | `pom.xml:73` |
| `paimon.version` | 1.3.1 | `pom.xml:93` |
| CI `JDK_VERSION` | 11 | `.github/workflows/mvn-ci-build.yml:36` |

The default was never moved to Spark 3.5 either. #671 was the last attempt and it was closed as stale in April 2026.

The `scala-2.13` profile at `pom.xml:1017` exists but is not usable as written:

- It is still gated by the comment `` (`pom.xml:1016`), which is now stale. `hudi-utilities_2.13` and `hudi-spark4.0-bundle_2.13` both publish at 1.2.0.
- Its only body is a `pluginManagement` block configuring `scala-maven-plugin` with `-target:jvm-1.8` (`pom.xml:1038`). That configuration is inert: the repository contains **zero `.scala` sources**, and no module declares `scala-maven-plugin` under ``, so the plugin never executes. The `-target:jvm-1.8` argument is therefore not a real blocker, only dead configuration to remove.

The practical effect is that the profile only swaps `scala.binary.version`, which is all that is actually needed, since every Scala-suffixed dependency is templated on that property.

## Upstream artifacts

All required Spark 4.0 artifacts are published. Nothing external blocks this work. Verified on Maven Central:

| Artifact | Versions available |
|---|---|
| `org.apache.hudi:hudi-spark4.0-bundle_2.13` | 1.1.0, 1.1.1, 1.2.0 |
| `org.apache.hudi:hudi-utilities_2.13` | up to 1.2.0 |
| `org.apache.iceberg:iceberg-spark-runtime-4.0_2.13` | 1.10.0, 1.10.1, 1.10.2, 1.11.0 |
| `io.delta:delta-spark_2.13` | 4.0.0 |
| `org.apache.paimon:paimon-spark-4.0` | 1.3.0, 1.3.1, 1.3.2 |
| `io.delta:delta-standalone_2.13` | 3.3.0 and later |
| `io.delta:delta-hive_2.13` | 3.0.0 and later |

The last two matter because `xtable-hive-metastore` pins `delta.standalone.version` 3.3.0 and `delta.hive.version` 3.0.0 (`pom.xml:98-99`). Both cross-build for 2.13, so that module does not block the Scala flip.

One convenient detail: Hudi, Iceberg and Paimon all use `4.0` as their Spark qualifier, matching the existing artifactId templates `hudi-spark${spark.version.prefix}-bundle_${scala.binary.version}`, `iceberg-spark-runtime-${spark.version.prefix}_${scala.binary.version}` and `paimon-spark-${spark.version.prefix}`. Setting `spark.version.prefix` to `4.0` covers all three with no artifactId edits.

## Build work

1. **Flip the default build to Scala 2.13.** Spark 4.0 only publishes for 2.13. Make `scala-2.13` the `activeByDefault` profile, drop the stale Hudi comment, and delete the inert `scala-maven-plugin` block.
2. **Raise the compiler target off Java 8.** Spark 4.0 requires Java 17. Set `javaVersion` to 17, and raise `JDK_VERSION` in `mvn-ci-build.yml` and `mvn-license-check.yml`, plus the hardcoded `java-version: '11'` in `package-deploy.yml:31` and `source-build-check.yml:50`. The release profile also hardcodes `1.8` for the javadoc plugin at `pom.xml:1120`. The `quarkus.platform.version` pin at 3.2.12.Final carries the comment `` and should be re-evaluated for `xtable-service`.
3. **Rename the Delta Spark artifact.** Delta renamed `delta-core` to `delta-spark` in 3.0, so `io.delta:delta-core_${scala.binary.version}` (`pom.xml:331`) becomes `delta-spark_${scala.binary.version}` at 4.0.0. Four consumers reference the old coordinate: `xtable-core`, `xtable-service`, `xtable-hudi-support-extensions`, and `xtable-utilities`, where it also appears as a shade ``.
4. **Bump Iceberg** from 1.9.2 to 1.10.0 or later, the first release with Spark 4.0 artifacts.
5. **Decide the Spark 3.5 step.** The list preference was to fold 3.5 into this upgrade rather than land it separately. A single intermediate commit setting the prefix to `3.5` may still be useful to isolate Delta 2.4 to 3.x breakage from the Scala and Java flip.

## Code work

This is not a version-bump-only change. Two areas need real edits.

**Delta internal Scala APIs.** The Delta reader and writer call Delta's internal Scala classes directly from Java, not the public `io.delta.tables` API. Across 18 files (9 in `src/main`, 9 in tests) the imports include `org.apache.spark.sql.delta.DeltaLog`, `Snapshot`, `OptimisticTransaction`, `DeltaHistoryManager`, `DeltaConfigs`, `DeltaOperations`, `GeneratedColumn`, and `actions.{AddFile, RemoveFile, Metadata, Action, CommitInfo, Format, DeletionVectorDescriptor}`. These carry no compatibility guarantee and change between Delta 2.4 and 4.0. Expect this to be the bulk of the work. Main-source files affected:

- `DeltaActionsConverter`, `DeltaConversionSource`, `DeltaConversionTarget`, `DeltaConversionUtils`, `DeltaDataFileExtractor`, `DeltaDataFileUpdatesExtractor`, `DeltaIncrementalChangesState`, `DeltaStatsExtractor`, `DeltaTableExtractor`

This is a second, code-level argument for landing #886 first. The Kernel path uses the stable `io.delta.kernel` API instead, and `delta.kernel.version` is already at 4.0.0 (`pom.xml:60`). The more of the Delta path that runs through Kernel before this upgrade starts, the smaller the internal-API surface left to port.

**Scala 2.12 to 2.13 interop from Java.** 15 Java files import `scala.*` directly, centered on `ScalaUtils.java` and the Delta and Kernel packages. The two risk points are `scala.collection.JavaConverters` (11 usages), deprecated in 2.13 in favour of `scala.jdk.javaapi.CollectionConverters`, and `scala.collection.Seq` (6 usages), whose relationship to `scala.Seq` changed in 2.13.

## Scope

- [ ] Make `scala-2.13` the default profile; remove the stale Hudi comment and the inert `scala-maven-plugin` configuration.
- [ ] Set `javaVersion` to 17; update all four workflows and the javadoc ``; re-evaluate the Quarkus pin.
- [ ] Set `spark.version` to 4.0.x and `spark.version.prefix` to `4.0`.
- [ ] Move `delta-core_` to `delta-spark_` and `delta.version` to 4.0.0, in all four consuming modules including the shade includes.
- [ ] Bump `iceberg.version` to 1.10.0 or later.
- [ ] Port the Delta internal-API call sites across the 18 affected files.
- [ ] Fix the Scala 2.13 interop in the 15 files importing `scala.*`.
- [ ] Confirm `xtable-hive-metastore` builds on 2.13 with the current `delta-standalone` and `delta-hive` pins.
- [ ] Run the full integration suite, `ITConversionController` in particular, on Spark 4.0.
- [ ] Decide whether to keep an intermediate Spark 3.5 commit.

## Open questions

- Do we keep a Spark 3.4 or 3.5 profile for users who cannot move to Java 17, or is the default the only supported combination? A cross-build matrix in CI would roughly double build time.
- Does the `xtable-spark-runtime` work in #836 need to land first, so the bundle is built once against the new baseline rather than twice?

## Related

- #886, Delta Kernel as the default Delta path. Strongly preferred to land before this.
- #836, `xtable-spark-runtime`.
- #671, the closed Spark 3.5 attempt.
- #896, shrink the `xtable-utilities` bundled jar, which touches the same shade configuration.

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Start with the version and profile properties in pom.xml, then inspect the listed Delta call sites, ScalaUtils.java, and the four workflow files. Review the 18 Delta files and 15 Scala interop files before changing dependencies. Done means the default Spark 4.0/Scala 2.13/Java 17 build passes, xtable-hive-metastore builds, and the full integration suite including ITConversionController succeeds.

Written by the indexing model from the issue text.