apache / apache/incubator-xtable
Introduce xtable-spark-runtime: thin, relocated Spark bundle for in-job metadata sync
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 212
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 16
Description
### Motivation
XTable's modules are published to Maven, so a user can depend on
`xtable-core` and assemble their own runtime today. But there is no
maintained, thin, drop-in artifact — everyone re-solves the same
shading/classpath problem, or uses `xtable-utilities`, an unshaded ~1GB fat
jar that isn't practical to add to a Spark job and requires running a
separate sync process with multiple config files.
XTable conversion is metadata-only and lightweight, so the common case —
"I already write this table with Spark, keep it in sync in other formats" —
should be a one-dependency, config-only addition to an existing pipeline.
### Proposal
A new module `xtable-spark-runtime` producing a thin, relocated bundle that:
- is added via `--jars` / `--packages`,
- is activated through config only (no user code change),
- runs incremental `ConversionController.sync(...)` on the driver after each
successful write to a source table.
**Usage**
```
spark-submit --packages org.apache.xtable:xtable-spark-runtime_2.12: \
--conf spark.sql.queryExecutionListeners=org.apache.xtable.spark.XTableSyncListener \
--conf spark.xtable.tables=/warehouse/db/orders \
--conf spark.xtable.orders.sourceFormat=HUDI \
--conf spark.xtable.orders.targets=ICEBERG,DELTA
```
### Packaging (Hudi/Iceberg model)
- Spark / Hadoop → `provided` (on the cluster, not bundled).
- Curated dependency allowlist (no bundling of the full transitive closure).
- Relocate `guava` / `jackson` / `protobuf` / `avro` / `commons` under
`org.apache.xtable.shaded.*` to avoid classpath clashes with the cluster.
- Target size: tens of MB.
### Design
- `XTableSyncListener implements QueryExecutionListener` (batch) and a
`StreamingQueryListener` variant.
- On write-success: resolve target table(s) from `spark.xtable.*`, submit an
incremental sync to a debounced, single-flight-per-table driver executor.
- Sync is idempotent + incremental, so a missed trigger self-heals on the
next commit.
### Scope / non-goals
- No data rewriting — metadata translation only.
- Does not replace the standalone CLI (`RunSync`); complements it.
### Open questions
- [ ] Table selection: config-declared (proposed) vs. plan-inspection auto-detect.
- [ ] Batch jobs: async (needs shutdown-hook drain) vs. blocking.
- [ ] Also ship a `CALL xtable.sync(...)` SQL procedure in the same jar?
### Tasks
- [ ] Scaffold `xtable-spark-runtime` module (pom: provided engines, allowlist, relocations).
- [ ] Implement `XTableSyncListener` + `spark.xtable.*` config parsing.
- [ ] Debounced driver executor + shutdown drain.
- [ ] `ITXTableSyncListener` (embedded `local[*]`, Hudi→Delta/Iceberg).
- [ ] spark-shell smoke test + README section.
- [ ] (optional) `CALL xtable.sync(...)` procedure.
### Related
- DISCUSS thread: https://lists.apache.org/thread/w9lql69ovy0rd8mkljtly9f7z7kyb751
- Bundle/licensing history: dev@ thread "[DISCUSS] Excluding xtable-utilities from the release process", #536.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the existing ConversionController.sync(...) and RunSync context, then review the proposed module pom and the README conventions. Use the ITXTableSyncListener task and spark-shell smoke test as the validation entry points. Done means the thin relocated runtime, listeners, configuration, debounced executor, integration test, and documentation are implemented after resolving the listed design questions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100