apache / apache/incubator-xtable

Introduce xtable-spark-runtime: thin, relocated Spark bundle for in-job metadata sync

Open
#836 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
1.2k
Forks
212
Avg merge
4d 9h
Merged PRs (30d)
16

Description

### Motivation

XTable's modules are published to Maven, so a user can depend on
`xtable-core` and assemble their own runtime today. But there is no
maintained, thin, drop-in artifact — everyone re-solves the same
shading/classpath problem, or uses `xtable-utilities`, an unshaded ~1GB fat
jar that isn't practical to add to a Spark job and requires running a
separate sync process with multiple config files.

XTable conversion is metadata-only and lightweight, so the common case —
"I already write this table with Spark, keep it in sync in other formats" —
should be a one-dependency, config-only addition to an existing pipeline.

### Proposal

A new module `xtable-spark-runtime` producing a thin, relocated bundle that:

- is added via `--jars` / `--packages`,
- is activated through config only (no user code change),
- runs incremental `ConversionController.sync(...)` on the driver after each
successful write to a source table.

**Usage**
```
spark-submit --packages org.apache.xtable:xtable-spark-runtime_2.12: \
--conf spark.sql.queryExecutionListeners=org.apache.xtable.spark.XTableSyncListener \
--conf spark.xtable.tables=/warehouse/db/orders \
--conf spark.xtable.orders.sourceFormat=HUDI \
--conf spark.xtable.orders.targets=ICEBERG,DELTA
```

### Packaging (Hudi/Iceberg model)
- Spark / Hadoop → `provided` (on the cluster, not bundled).
- Curated dependency allowlist (no bundling of the full transitive closure).
- Relocate `guava` / `jackson` / `protobuf` / `avro` / `commons` under
`org.apache.xtable.shaded.*` to avoid classpath clashes with the cluster.
- Target size: tens of MB.

### Design
- `XTableSyncListener implements QueryExecutionListener` (batch) and a
`StreamingQueryListener` variant.
- On write-success: resolve target table(s) from `spark.xtable.*`, submit an
incremental sync to a debounced, single-flight-per-table driver executor.
- Sync is idempotent + incremental, so a missed trigger self-heals on the
next commit.

### Scope / non-goals
- No data rewriting — metadata translation only.
- Does not replace the standalone CLI (`RunSync`); complements it.

### Open questions
- [ ] Table selection: config-declared (proposed) vs. plan-inspection auto-detect.
- [ ] Batch jobs: async (needs shutdown-hook drain) vs. blocking.
- [ ] Also ship a `CALL xtable.sync(...)` SQL procedure in the same jar?

### Tasks
- [ ] Scaffold `xtable-spark-runtime` module (pom: provided engines, allowlist, relocations).
- [ ] Implement `XTableSyncListener` + `spark.xtable.*` config parsing.
- [ ] Debounced driver executor + shutdown drain.
- [ ] `ITXTableSyncListener` (embedded `local[*]`, Hudi→Delta/Iceberg).
- [ ] spark-shell smoke test + README section.
- [ ] (optional) `CALL xtable.sync(...)` procedure.

### Related
- DISCUSS thread: https://lists.apache.org/thread/w9lql69ovy0rd8mkljtly9f7z7kyb751
- Bundle/licensing history: dev@ thread "[DISCUSS] Excluding xtable-utilities from the release process", #536.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the existing ConversionController.sync(...) and RunSync context, then review the proposed module pom and the README conventions. Use the ITXTableSyncListener task and spark-shell smoke test as the validation entry points. Done means the thin relocated runtime, listeners, configuration, debounced executor, integration test, and documentation are implemented after resolving the listed design questions.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.