Spark built in connector for Druid
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Motivation
In my current project there is a requirement to load HDFS data combined with Db2 data which is huge in number. The data volume is almost 1 TB per day. We need to join these two huge data sets and the perform business logic on top if to create daily report for insights.
### Proposed changes
Today we need to use Tranquilizer Beam, storm or any other middle ware with spark to push the data into Druid. If we have spark connector directly connection we can all the transformation directly into spark push it like we have kafka indexing services.
I am not very professional in Java tech stack but learning , hopefully if no is working on this proposal one day I will create it.:)
Thanks,
Ashish
Contributor guide
Research direction
The issue names a proposed Spark connector for loading data into Druid, involving HDFS and Db2, but it does not identify files, tests, or an entry point. Start by reviewing the project's connector and ingestion architecture, then define supported inputs, transformations, and an acceptance test for direct Spark-to-Druid loading.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 18/100