[VL] Delta Lake write
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
To add write support for Delta Lake in Velox backend.
Work items:
1. Add native write support for Spark 3.5 + Delta 3.3
https://github.com/apache/incubator-gluten/pull/10802
https://github.com/apache/incubator-gluten/pull/10801
3. (TODO) Add native write support for Spark 4.0 + Delta 4.0
4. (PR pending) Native statistics tracker to avoid C2R overhead
https://github.com/apache/incubator-gluten/pull/11419
5. (TODO, optional) Add native write support for lower versions (Spark 3.4 + Delta 2.4)
6. Offload [`DeltaOptimizedWriterExec`](https://github.com/delta-io/delta/blob/4b2beb096017d813475f692270bc20c113c0d974/spark/src/main/scala/org/apache/spark/sql/delta/perf/DeltaOptimizedWriterExec.scala) for optimize options
https://github.com/apache/incubator-gluten/pull/11461
---
PoC at #10216.
Gaps so far:
1. Constraint expression needs to offload (C2R / R2C added otherwise).
2. The overwritten Delta classes in Gluten should be avoided (`DeltaParquetFileFormat` for example).
3. Copied test code from Delta, if there's a way to avoid the practice.
Contributor guide
Assessment
This issue has not been assessed yet.