apache / apache/incubator-xtable
Docs: Delta Kernel is not auto-selected for in-job syncs, only for the XTableSparkSync CLI
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 212
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 16
Description
`website/docs/how-to-spark-runtime.md` tells readers that the Delta implementation is chosen for
them on Spark 3.5:
> Delta is the only engine whose implementation depends on the Spark version, and the jar chooses
> the right one automatically
> | 3.5.x and newer | ✅ | Delta Kernel, selected automatically |
> Delta Standalone doesn't run on Spark 3.5, so on 3.5 and newer a Delta source or target is
> routed through Delta Kernel with no flag needed.
That holds for the CLI entry point only. In `XTableSparkSync.main`, on the `0.4.0-incubating` tag:
```java
boolean deltaInvolved =
TableFormat.DELTA.equals(sourceFormat) || targetFormats.contains(TableFormat.DELTA);
boolean useDeltaKernel = useDeltaKernelFlag;
if (!useDeltaKernel && deltaInvolved && isSparkAtLeast35(spark.version())) {
useDeltaKernel = true;
...
}
```
The in-job API has no such logic. `XTableSyncService.sync` reads the flag straight off the spec:
```java
ConversionSourceProvider sourceProvider =
sourceProviderFor(spec.getSourceFormat(), spec.isUseDeltaKernel());
```
and `TableSyncSpec.useDeltaKernel` is a plain `boolean` that defaults to `false`. Nothing in
`xtable-spark-runtime` inspects the Spark version outside `XTableSparkSync`; grepping the module
on the tag returns `isSparkAtLeast35` as the only call site.
### Impact
The page leads with the embedded API ("This is what the jar is for") and its example syncs to
`DELTA`. A reader who copies that example onto Spark 3.5 gets no Kernel auto-selection, falls
through to `DeltaConversionSourceProvider`, and hits the `delta-core` path that the same page says
does not run on 3.5. The one sentence that does mention `.useDeltaKernel(true)` frames it as a
Spark 3.4 opt-in, which points the reader away from the case where it is actually required.
### Suggested fix
Separate the two entry points in the "Spark version support" section: `XTableSparkSync`
auto-enables Kernel when Delta is involved and Spark is 3.5 or later, while `XTableSyncService`
requires `.useDeltaKernel(true)` explicitly on Spark 3.5.
Split out of #920 so that issue stays scoped to the review comments on #913.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with website/docs/how-to-spark-runtime.md and its “Spark version support” section, then compare the documented behavior with XTableSparkSync.main and XTableSyncService.sync. Clarify the separate entry points and state when the embedded API requires useDeltaKernel(true); done when the page no longer implies automatic selection for both APIs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 85/100