apache / apache/gluten

[VL] Delta Lake write

Open
#10,215 2 comments 0 reactions 1 assignee Claimed by @zhztheplayer View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

To add write support for Delta Lake in Velox backend.

Work items:

1. Add native write support for Spark 3.5 + Delta 3.3
https://github.com/apache/incubator-gluten/pull/10802
https://github.com/apache/incubator-gluten/pull/10801
3. (TODO) Add native write support for Spark 4.0 + Delta 4.0
4. (PR pending) Native statistics tracker to avoid C2R overhead
https://github.com/apache/incubator-gluten/pull/11419
5. (TODO, optional) Add native write support for lower versions (Spark 3.4 + Delta 2.4)
6. Offload [`DeltaOptimizedWriterExec`](https://github.com/delta-io/delta/blob/4b2beb096017d813475f692270bc20c113c0d974/spark/src/main/scala/org/apache/spark/sql/delta/perf/DeltaOptimizedWriterExec.scala) for optimize options
https://github.com/apache/incubator-gluten/pull/11461

---

PoC at #10216.

Gaps so far:

1. Constraint expression needs to offload (C2R / R2C added otherwise).
2. The overwritten Delta classes in Gluten should be avoided (`DeltaParquetFileFormat` for example).
3. Copied test code from Delta, if there's a way to avoid the practice.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.