51zero / 51zero/eel-sdk

Support Distributed writes with EEL

Open
#253 3 comments 0 reactions 1 assignee Claimed by @hannesmiller View on GitHub
enhancement
Dominant language
Scala
Stars
147
Forks
32
PR merge metrics
No merged PRs in 30d

Description

Support Distributed writes with EEL

- N writers via JdbcSource -> KafkaSink
- N Writers via HiveSink/KuduSink/HBaseSink
- Now what if the **HiveSink** and others that use a **LinkedBlockingQueue** to service multiple writer threads could do this in a distributed fashion by wrapping the **LinkedBlockingQueue** interface, i.e. an implementation that wraps a Kafka topic - default one would still remain as threads?
- The gotcha is that when you are out-of-process you lose control on how to partition the data into reasonable sizes
- However for row oriented storage systems like Kudu and HBase it's perfect - the same usage pattern would even work for the JdbcSink

What do you think?

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue proposes distributed writes by replacing LinkedBlockingQueue with a Kafka topic wrapper for sinks like HiveSink, KuduSink, HBaseSink, and JdbcSink. Start by examining the sink implementations in the codebase, particularly the queueing mechanism. Understand how partitioning works for row-oriented storage systems. The goal is to design a distributed queue interface that maintains data partitioning control.

Written by the indexing model from the issue text.

Assessment

Tech stack
kafka, scala
Domain
backend, data-engineering, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.