Support Distributed writes with EEL

Support Distributed writes with EEL

- N writers via JdbcSource -> KafkaSink
- N Writers via HiveSink/KuduSink/HBaseSink
- Now what if the **HiveSink** and others that use a **LinkedBlockingQueue** to service multiple writer threads could do this in a distributed fashion by wrapping the **LinkedBlockingQueue** interface, i.e. an implementation that wraps a Kafka topic - default one would still remain as threads?
- The gotcha is that when you are out-of-process you lose control on how to partition the data into reasonable sizes
- However for row oriented storage systems like Kudu and HBase it's perfect - the same usage pattern would even work for the JdbcSink

What do you think?

The issue proposes distributed writes by replacing LinkedBlockingQueue with a Kafka topic wrapper for sinks like HiveSink, KuduSink, HBaseSink, and JdbcSink. Start by examining the sink implementations in the codebase, particularly the queueing mechanism. Understand how partitioning works for row-oriented storage systems. The goal is to design a distributed queue interface that maintains data partitioning control.

Rédigé par le modèle d'indexation à partir du texte de l'issue.