AbsaOSS / AbsaOSS/enceladus

Dynamic conformance on streaming sources

Aperta
#1,013 0 commenti 0 reazioni 1 assegnatario Rivendicata da @yruslan Vedi su GitHub
Epic feature priority: high Streaming
Lingua principale
Scala
Stelle
33
Fork
16
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

## Background
_(Based on the investigation done in #897)_

Conforming streaming sources is something we would like to have. Although Spark API is similar for batch and stream processing, they are not the same. There is a number of issues we identified when trying to run Conformance on a streaming source.

## Epic
We need to investigate possibilities and implement missing pieces in order to support streaming sources in DCE jobs.
* Implement an ability to run DCE on streaming sources. Add configuration parameters for streaming jobs, like a checkpoint location, etc.
* Implement a streaming-friendly conformance mapping rule. Possibly use a mapping table broadcast approach.
* Implement a streaming-friendly control framework (in Atum).

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.