Implement WriteFilesExecTransformer discussion
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
Since Spark 3.4 introduced the WriteFilesExec operator to facilitate write operations in [SPARK-41708](https://issues.apache.org/jira/browse/SPARK-41708), we can utilize the WriteFilesExec operator to implement native Parquet write. This can be achieved by adding a WriteFilesExecTransformer and offloading the process to the Velox TableWriteNode. To achieve this, consider the following implementation steps:
1. Add `WriteFilesRelNode`.
2. Introduce the `WriteFilesExecTransformer `operator to utilize the native `TableWriteNode `for writing data into files.
3. Introduce the `ColumnarWriteFile `operator by inheriting the `WriteFilesExec `class. Implement the `doExecuteWrite()` method within the `ColumnarWriteFile `class to retrieve the `RDD[WriterCommitMessage] `by jni call.
Do you have any input?@ulysses-you @zhztheplayer @zhouyuan
Contributor guide
Assessment
This issue has not been assessed yet.