apache / apache/druid

Spark built in connector for Druid

Open
#9,835 1 comment 0 reactions 0 assignees View on GitHub
Apache Spark Design Review Proposal
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Motivation
In my current project there is a requirement to load HDFS data combined with Db2 data which is huge in number. The data volume is almost 1 TB per day. We need to join these two huge data sets and the perform business logic on top if to create daily report for insights.

### Proposed changes
Today we need to use Tranquilizer Beam, storm or any other middle ware with spark to push the data into Druid. If we have spark connector directly connection we can all the transformation directly into spark push it like we have kafka indexing services.

I am not very professional in Java tech stack but learning , hopefully if no is working on this proposal one day I will create it.:)

Thanks,
Ashish

Contributor guide

Open the contributing guide

Research direction

The issue names a proposed Spark connector for loading data into Druid, involving HDFS and Db2, but it does not identify files, tests, or an entry point. Start by reviewing the project's connector and ingestion architecture, then define supported inputs, transformations, and an acceptance test for direct Spark-to-Druid loading.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
18/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.