apache / apache/incubator-xtable

Docs: Delta Kernel is not auto-selected for in-job syncs, only for the XTableSparkSync CLI

Open Beginner friendly
#921 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
1.2k
Forks
212
Avg merge
4d 9h
Merged PRs (30d)
16

Description

`website/docs/how-to-spark-runtime.md` tells readers that the Delta implementation is chosen for
them on Spark 3.5:

> Delta is the only engine whose implementation depends on the Spark version, and the jar chooses
> the right one automatically

> | 3.5.x and newer | ✅ | Delta Kernel, selected automatically |

> Delta Standalone doesn't run on Spark 3.5, so on 3.5 and newer a Delta source or target is
> routed through Delta Kernel with no flag needed.

That holds for the CLI entry point only. In `XTableSparkSync.main`, on the `0.4.0-incubating` tag:

```java
boolean deltaInvolved =
TableFormat.DELTA.equals(sourceFormat) || targetFormats.contains(TableFormat.DELTA);
boolean useDeltaKernel = useDeltaKernelFlag;
if (!useDeltaKernel && deltaInvolved && isSparkAtLeast35(spark.version())) {
useDeltaKernel = true;
...
}
```

The in-job API has no such logic. `XTableSyncService.sync` reads the flag straight off the spec:

```java
ConversionSourceProvider sourceProvider =
sourceProviderFor(spec.getSourceFormat(), spec.isUseDeltaKernel());
```

and `TableSyncSpec.useDeltaKernel` is a plain `boolean` that defaults to `false`. Nothing in
`xtable-spark-runtime` inspects the Spark version outside `XTableSparkSync`; grepping the module
on the tag returns `isSparkAtLeast35` as the only call site.

### Impact

The page leads with the embedded API ("This is what the jar is for") and its example syncs to
`DELTA`. A reader who copies that example onto Spark 3.5 gets no Kernel auto-selection, falls
through to `DeltaConversionSourceProvider`, and hits the `delta-core` path that the same page says
does not run on 3.5. The one sentence that does mention `.useDeltaKernel(true)` frames it as a
Spark 3.4 opt-in, which points the reader away from the case where it is actually required.

### Suggested fix

Separate the two entry points in the "Spark version support" section: `XTableSparkSync`
auto-enables Kernel when Delta is involved and Spark is 3.5 or later, while `XTableSyncService`
requires `.useDeltaKernel(true)` explicitly on Spark 3.5.

Split out of #920 so that issue stays scoped to the review comments on #913.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with website/docs/how-to-spark-runtime.md and its “Spark version support” section, then compare the documented behavior with XTableSparkSync.main and XTableSyncService.sync. Clarify the separate entry points and state when the embedded API requires useDeltaKernel(true); done when the page no longer implies automatic selection for both APIs.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
85/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.