apache / apache/incubator-xtable

Set delta kernel as the default path for 0.5.0 release

Open
#886 10 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
1.2k
Forks
212
Avg merge
4d 9h
Merged PRs (30d)
16

Description

### Feature Request / Improvement

The Delta Kernel conversion source and target are on `main` (#729, #801), but they are opt-in. A Delta sync only routes through them when the target table sets `xtable.delta.target.use_kernel=true`, which `DeltaConversionTargetConfig` reads and `ConversionTargetFactory` dispatches on. The default is `false`, so the default Delta path today is still Delta Standalone.

This issue tracks making Kernel the default for the 0.5.0 release, together with the work that has to land before that flip is safe.

Kernel matters beyond parity. It removes the last hard Spark dependency in the sync path, since the Hudi and Iceberg paths are already pure Java and Delta was the one format anchoring us to `delta-spark` and `DeltaLog`.

**Current state**

- `xtable.delta.target.use_kernel` defaults to `false`. Both implementations are registered under `TableFormat.DELTA` in the same `META-INF/services` file, and `ConversionTargetFactory` picks between them on that flag.
- The Kernel path is covered only by its own suites: `ITDeltaKernelConversionSource`, `TestDeltaKernelReadWriteIntegration`, `TestDeltaKernelSync`, plus the extractor, partition, schema and stats unit tests.
- `ITConversionController` is where every other source and target combination is validated for dataset equivalence across sync modes and partitioning. It has **no** Kernel coverage. Its `generateTestParametersForFormatsSyncModesAndPartitioning()` iterates `HUDI, DELTA, ICEBERG, PAIMON`, and `getConversionSourceProvider(String)` returns `DeltaConversionSourceProvider` for `DELTA` with no Kernel branch. `DeltaKernelConversionSourceProvider` is not even imported there.

**Prerequisites, before the default changes**

1. Cover Kernel in `ITConversionController`, on both sides:
- as a **source**, so `DeltaKernelConversionSourceProvider` runs the same equivalence checks `DeltaConversionSourceProvider` runs today, across both sync modes and both partitioning cases;
- as a **target**, by setting `xtable.delta.target.use_kernel=true` for the Kernel variant, so both writers are validated.

Keep the Standalone cases as they are. The goal is both implementations passing the same assertions, not replacing one with the other. If doubling the whole matrix costs too much wall-clock time, a separate parameter or a targeted subset is fine, as long as both sync modes and both partitioning cases are covered for each of source and target.

2. Re-verify the Delta conversion issues reported recently against the Kernel path, so the flip does not quietly regress fixes that only exist on the Standalone path: #711 (column rename during schema sync), #779 (log truncation on Delta to Iceberg incremental sync), #813 (`_delta_log` treated as a partition), #860 (snapshot reloaded on every commit), #586, #641.

3. Answer feature parity for deletion vectors (#713). If Kernel cannot express something Standalone can, that needs to be known and documented before it becomes the default rather than after.

**The change itself**

- Flip the default of `xtable.delta.target.use_kernel` to `true`.
- Keep the flag as an escape hatch for one release, so a user who hits a regression can go back to Standalone without downgrading.
- Call the change out in the release notes, with the flag documented as the way to opt out.

**Out of scope**

The packaging and module questions raised on the same dev@ thread. The Kernel path being solid is a precondition there too, but the packaging decision is independent of this issue.

Context from the dev@ discussion, where the point was made that the dedicated suites are not the right bar and the shared harness is:
https://lists.apache.org/thread/bx1xyh0ml30r60od1hdvhndyd16b375j

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with ITConversionController, especially generateTestParametersForFormatsSyncModesAndPartitioning() and getConversionSourceProvider(String), then review the listed Delta Kernel suites and DeltaConversionTargetConfig. Verify both source and target coverage across sync modes and partitioning, recheck issues #711, #779, #813, #860, #586, #641 and deletion-vector parity, then confirm the default, escape hatch, and release-note documentation are covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data, release, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.