apache / apache/datafusion-comet
Explore integration with Delta Lake
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 190
Description
### What is the problem the feature request solves?
Comet currently only support either Spark's built-in data sources, or Iceberg ([WIP](https://github.com/apache/iceberg/pull/9841)). We should also consider supporting [Delta Lake](https://delta.io/) in future especially given it already has a Rust implementation [`delta-rs`](https://github.com/delta-io/delta-rs). To achieve that, however, we may need to first move away from our hybrid Parquet reader implementation to a fully native one.
cc @dennyglee per our discussion
### Describe the potential solution
Integrate Comet with `delta-rs`, so Spark queries reading from Delta Lake tables can also leverage Comet native execution.
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.