51zero / 51zero/eel-sdk

Support Distributed writes with EEL

未关闭
#253 3 条评论 0 个 reaction 已指派 1 人 已被 @hannesmiller 认领 在 GitHub 查看
enhancement
主要语言
Scala
星标
147
派生
32
PR 合并指标
30 天内没有已合并 PR

描述

Support Distributed writes with EEL

- N writers via JdbcSource -> KafkaSink
- N Writers via HiveSink/KuduSink/HBaseSink
- Now what if the **HiveSink** and others that use a **LinkedBlockingQueue** to service multiple writer threads could do this in a distributed fashion by wrapping the **LinkedBlockingQueue** interface, i.e. an implementation that wraps a Kafka topic - default one would still remain as threads?
- The gotcha is that when you are out-of-process you lose control on how to partition the data into reasonable sizes
- However for row oriented storage systems like Kudu and HBase it's perfect - the same usage pattern would even work for the JdbcSink

What do you think?

贡献指南

这个仓库没有索引到贡献指南

调研方向

The issue proposes distributed writes by replacing LinkedBlockingQueue with a Kafka topic wrapper for sinks like HiveSink, KuduSink, HBaseSink, and JdbcSink. Start by examining the sink implementations in the codebase, particularly the queueing mechanism. Understand how partitioning works for row-oriented storage systems. The goal is to design a distributed queue interface that maintains data partitioning control.

由索引模型根据 Issue 内容生成。

评估

技术栈
kafka, scala
领域
backend, data-engineering, stream-processing
Issue 类型
功能
难度
5/5
预计耗时
一周以上
活跃度
停滞
描述清晰度
需要澄清
新手友好度
20/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。