apache / apache/datafusion-comet

ZeroCopy Conversion from Spark ColumnarBatch

Open
#3,518 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

When spark DataSource api can read data as ColumnarBatch objects datafusion can already do a direct columnar to columnar conversion. This avoids row conversion, but still means data needs to be copied even when the data in ColumnarBatch is already arrow.

### Describe the potential solution

Conversion should first check if vectors in ColumnarBatch are instances of `org.apache.spark.sql.vectorized.ArrowColumnVector`. When that's true, conversion should be zero-copy and probably ignore datafusion's maxRowsPerBatch config (??).

### Additional context

I'm working on [spark-adbc](https://github.com/tokoko/spark-adbc) data source. data source outputs `ArrowColumnVector`s in ColumnarBatch. while using comet helps avoid row conversion right after read, there's still some penalty for copying vectors over.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.