apache / apache/gluten

Implement WriteFilesExecTransformer discussion

Open
#3,547 9 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

Since Spark 3.4 introduced the WriteFilesExec operator to facilitate write operations in [SPARK-41708](https://issues.apache.org/jira/browse/SPARK-41708), we can utilize the WriteFilesExec operator to implement native Parquet write. This can be achieved by adding a WriteFilesExecTransformer and offloading the process to the Velox TableWriteNode. To achieve this, consider the following implementation steps:

1. Add `WriteFilesRelNode`.
2. Introduce the `WriteFilesExecTransformer `operator to utilize the native `TableWriteNode `for writing data into files.
3. Introduce the `ColumnarWriteFile `operator by inheriting the `WriteFilesExec `class. Implement the `doExecuteWrite()` method within the `ColumnarWriteFile `class to retrieve the `RDD[WriterCommitMessage] `by jni call.

Do you have any input?@ulysses-you @zhztheplayer @zhouyuan

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.