apache / apache/datafusion-comet

Add native_datafusion V2 DataSource API reader

Open
#3,481 0 comments 0 reactions 1 assignee Claimed by @mbutrovich View on GitHub
enhancement native_datafusion
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

I will do this after #3446 merges.

### What is the problem the feature request solves?

While working on #3446 I tested implementing a DataSource V2 compatible `native_datafusion` scan. I got tests passing, but then realized that Spark's DataSource V2 Parquet scan has fewer features than V1, such as [not supporting DPP](https://github.com/apache/spark/pull/52180). Maybe Spark implemented the V2 Parquet reader to dogfood the V2 Data Source API without external dependencies to test an API that was really created for things like Iceberg, Delta, etc.

However I think newer catalog implementations might return V2 DataSource API Parquet table references, so we should probably still support.

### Describe the potential solution

Implement a `CometNativeBatchScanExec` operator that converts Spark `BatchScanExec` (with `ParquetScan`)

This should hopefully serialize down to the same proto as `CometNativeScanExec` and handled transparently on the native side in planner.rs.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.