apache / apache/incubator-xtable
Set delta kernel as the default path for 0.5.0 release
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 212
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 16
Description
### Feature Request / Improvement
The Delta Kernel conversion source and target are on `main` (#729, #801), but they are opt-in. A Delta sync only routes through them when the target table sets `xtable.delta.target.use_kernel=true`, which `DeltaConversionTargetConfig` reads and `ConversionTargetFactory` dispatches on. The default is `false`, so the default Delta path today is still Delta Standalone.
This issue tracks making Kernel the default for the 0.5.0 release, together with the work that has to land before that flip is safe.
Kernel matters beyond parity. It removes the last hard Spark dependency in the sync path, since the Hudi and Iceberg paths are already pure Java and Delta was the one format anchoring us to `delta-spark` and `DeltaLog`.
**Current state**
- `xtable.delta.target.use_kernel` defaults to `false`. Both implementations are registered under `TableFormat.DELTA` in the same `META-INF/services` file, and `ConversionTargetFactory` picks between them on that flag.
- The Kernel path is covered only by its own suites: `ITDeltaKernelConversionSource`, `TestDeltaKernelReadWriteIntegration`, `TestDeltaKernelSync`, plus the extractor, partition, schema and stats unit tests.
- `ITConversionController` is where every other source and target combination is validated for dataset equivalence across sync modes and partitioning. It has **no** Kernel coverage. Its `generateTestParametersForFormatsSyncModesAndPartitioning()` iterates `HUDI, DELTA, ICEBERG, PAIMON`, and `getConversionSourceProvider(String)` returns `DeltaConversionSourceProvider` for `DELTA` with no Kernel branch. `DeltaKernelConversionSourceProvider` is not even imported there.
**Prerequisites, before the default changes**
1. Cover Kernel in `ITConversionController`, on both sides:
- as a **source**, so `DeltaKernelConversionSourceProvider` runs the same equivalence checks `DeltaConversionSourceProvider` runs today, across both sync modes and both partitioning cases;
- as a **target**, by setting `xtable.delta.target.use_kernel=true` for the Kernel variant, so both writers are validated.
Keep the Standalone cases as they are. The goal is both implementations passing the same assertions, not replacing one with the other. If doubling the whole matrix costs too much wall-clock time, a separate parameter or a targeted subset is fine, as long as both sync modes and both partitioning cases are covered for each of source and target.
2. Re-verify the Delta conversion issues reported recently against the Kernel path, so the flip does not quietly regress fixes that only exist on the Standalone path: #711 (column rename during schema sync), #779 (log truncation on Delta to Iceberg incremental sync), #813 (`_delta_log` treated as a partition), #860 (snapshot reloaded on every commit), #586, #641.
3. Answer feature parity for deletion vectors (#713). If Kernel cannot express something Standalone can, that needs to be known and documented before it becomes the default rather than after.
**The change itself**
- Flip the default of `xtable.delta.target.use_kernel` to `true`.
- Keep the flag as an escape hatch for one release, so a user who hits a regression can go back to Standalone without downgrading.
- Call the change out in the release notes, with the flag documented as the way to opt out.
**Out of scope**
The packaging and module questions raised on the same dev@ thread. The Kernel path being solid is a precondition there too, but the packaging decision is independent of this issue.
Context from the dev@ discussion, where the point was made that the dedicated suites are not the right bar and the shared harness is:
https://lists.apache.org/thread/bx1xyh0ml30r60od1hdvhndyd16b375j
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with ITConversionController, especially generateTestParametersForFormatsSyncModesAndPartitioning() and getConversionSourceProvider(String), then review the listed Delta Kernel suites and DeltaConversionTargetConfig. Verify both source and target coverage across sync modes and partitioning, recheck issues #711, #779, #813, #860, #586, #641 and deletion-vector parity, then confirm the default, escape hatch, and release-note documentation are covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data, release, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100