AbsaOSS / AbsaOSS/enceladus

Dynamic conformance on streaming sources

未关闭
#1,013 0 条评论 0 个 reaction 已指派 1 人 已被 @yruslan 认领 在 GitHub 查看
Epic feature priority: high Streaming
主要语言
Scala
星标
33
派生
16
PR 合并指标
30 天内没有已合并 PR

描述

## Background
_(Based on the investigation done in #897)_

Conforming streaming sources is something we would like to have. Although Spark API is similar for batch and stream processing, they are not the same. There is a number of issues we identified when trying to run Conformance on a streaming source.

## Epic
We need to investigate possibilities and implement missing pieces in order to support streaming sources in DCE jobs.
* Implement an ability to run DCE on streaming sources. Add configuration parameters for streaming jobs, like a checkpoint location, etc.
* Implement a streaming-friendly conformance mapping rule. Possibly use a mapping table broadcast approach.
* Implement a streaming-friendly control framework (in Atum).

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。