Dynamic conformance on streaming sources
- Ngôn ngữ chính
- Scala
- Star
- 33
- Fork
- 16
- Chỉ số merge pull request
- Không có pull request nào được merge trong 30 ngày
Mô tả
## Background
_(Based on the investigation done in #897)_
Conforming streaming sources is something we would like to have. Although Spark API is similar for batch and stream processing, they are not the same. There is a number of issues we identified when trying to run Conformance on a streaming source.
## Epic
We need to investigate possibilities and implement missing pieces in order to support streaming sources in DCE jobs.
* Implement an ability to run DCE on streaming sources. Add configuration parameters for streaming jobs, like a checkpoint location, etc.
* Implement a streaming-friendly conformance mapping rule. Possibly use a mapping table broadcast approach.
* Implement a streaming-friendly control framework (in Atum).
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.