apache / apache/datafusion-comet

Explore integration with Delta Lake

Open
#174 10 comments 11 reactions 1 assignee Claimed by @schenksj View on GitHub
enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 6h
Merged PRs (30d)
190

Description

### What is the problem the feature request solves?

Comet currently only support either Spark's built-in data sources, or Iceberg ([WIP](https://github.com/apache/iceberg/pull/9841)). We should also consider supporting [Delta Lake](https://delta.io/) in future especially given it already has a Rust implementation [`delta-rs`](https://github.com/delta-io/delta-rs). To achieve that, however, we may need to first move away from our hybrid Parquet reader implementation to a fully native one.

cc @dennyglee per our discussion

### Describe the potential solution

Integrate Comet with `delta-rs`, so Spark queries reading from Delta Lake tables can also leverage Comet native execution.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.