Support Distributed writes with EEL
- Lenguaje dominante
- Scala
- Estrellas
- 147
- Forks
- 32
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Descripción
Support Distributed writes with EEL
- N writers via JdbcSource -> KafkaSink
- N Writers via HiveSink/KuduSink/HBaseSink
- Now what if the **HiveSink** and others that use a **LinkedBlockingQueue** to service multiple writer threads could do this in a distributed fashion by wrapping the **LinkedBlockingQueue** interface, i.e. an implementation that wraps a Kafka topic - default one would still remain as threads?
- The gotcha is that when you are out-of-process you lose control on how to partition the data into reasonable sizes
- However for row oriented storage systems like Kudu and HBase it's perfect - the same usage pattern would even work for the JdbcSink
What do you think?
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Evaluación
Este issue todavía no se ha evaluado.