apache / apache/pinot

Batch Ingestion from Delta Table

Open
#9,084 2 comments 7 reactions 1 assignee Claimed by @athultr1997 View on GitHub
feature
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Currently, Pinot cannot perform batch ingestion from a [delta](https://delta.io/) table. It would be an excellent feature since the project is open-sourced and has many users.

We can build a new [RecordReader](https://github.com/apache/pinot/blob/6903856d399b4d7b09548a75a888382e1659e9e3/pinot-plugins/pinot-input-format/pinot-parquet/src/main/java/org/apache/pinot/plugin/inputformat/parquet/ParquetNativeRecordReader.java#L44) interface for delta tables utilizing the [delta-standalone](https://docs.delta.io/latest/delta-standalone.html) library.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.