apache / apache/druid

Delta IO Connector

Open
#12,598 0 comments 1 reaction 0 assignees View on GitHub
Design Review Proposal
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Motivation

Hello.

I have been using Apache Druid on and off for some time. Most recently, I used it as the base for a Lakehouse architecture I designed. I have been deploying it alone via K8S vanilla style, without any operators or hosted clouds etc with good success.
However, since I am a big fan of the open source data layer created by databricks: [delta.io](http://delta.io/) , I ended up doing tedious movements of delta -> raw parquet -> load to druid .

presto has recently created a [delta.io](http://delta.io/) adapter to link delta files directly, this is making me consider to use a presto cluster instead (would scale decently enough with current data load I have to work with).
However, is there anyone who has faced this issue and solved it in a “nice” way, or are there any plans to add a delta connector similar to what presto has done?

Here is a reference to what presto has added: https://prestodb.io/blog/2022/03/15/native-delta-lake-connector-for-presto
And a video: https://www.youtube.com/watch?v=JrXGkqpl7xk (fast forward to `21:40`).

### Proposed changes

Implement delta.io connector to load in into apache druid.

### Rationale

Currently it is very tedious, and a bit inefficient when you are working with delta.io as your underlaying layer, to then have another service extracting version `N` of delta to load into raw parquet, to make it accessible to `apache druid`.

### Operational impact

It should be done in a way so it does not impact backwards compatibility.

Contributor guide

Open the contributing guide

Research direction

Review Apache Druid's existing connector architecture and the linked Presto Delta Lake connector reference; the issue names no Druid files or tests. Establish the supported Delta Lake versions, read/version semantics, and backward-compatibility requirements before defining the work. Done would mean an agreed scope and a functioning Delta Lake connector integrated without breaking existing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.