apache / apache/datafusion-comet
ZeroCopy Conversion from Spark ColumnarBatch
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
When spark DataSource api can read data as ColumnarBatch objects datafusion can already do a direct columnar to columnar conversion. This avoids row conversion, but still means data needs to be copied even when the data in ColumnarBatch is already arrow.
### Describe the potential solution
Conversion should first check if vectors in ColumnarBatch are instances of `org.apache.spark.sql.vectorized.ArrowColumnVector`. When that's true, conversion should be zero-copy and probably ignore datafusion's maxRowsPerBatch config (??).
### Additional context
I'm working on [spark-adbc](https://github.com/tokoko/spark-adbc) data source. data source outputs `ArrowColumnVector`s in ColumnarBatch. while using comet helps avoid row conversion right after read, there's still some penalty for copying vectors over.
Contributor guide
Assessment
This issue has not been assessed yet.