Support Distributed writes with EEL
- Dominant language
- Scala
- Stars
- 147
- Forks
- 32
- PR merge metrics
- No merged PRs in 30d
Description
Support Distributed writes with EEL
- N writers via JdbcSource -> KafkaSink
- N Writers via HiveSink/KuduSink/HBaseSink
- Now what if the **HiveSink** and others that use a **LinkedBlockingQueue** to service multiple writer threads could do this in a distributed fashion by wrapping the **LinkedBlockingQueue** interface, i.e. an implementation that wraps a Kafka topic - default one would still remain as threads?
- The gotcha is that when you are out-of-process you lose control on how to partition the data into reasonable sizes
- However for row oriented storage systems like Kudu and HBase it's perfect - the same usage pattern would even work for the JdbcSink
What do you think?
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue proposes distributed writes by replacing LinkedBlockingQueue with a Kafka topic wrapper for sinks like HiveSink, KuduSink, HBaseSink, and JdbcSink. Start by examining the sink implementations in the codebase, particularly the queueing mechanism. Understand how partitioning works for row-oriented storage systems. The goal is to design a distributed queue interface that maintains data partitioning control.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kafka, scala
- Domain
- backend, data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100