SparkJobs return code based on data quality
オープン
Conformance
feature
priority: low
Standardization
under discussion
- 主要言語
- Scala
- スター
- 33
- フォーク
- 16
- PR マージ指標
- 30日以内にマージされた PR はありません
説明
## Background
`SparkJobs` return non-zero return code when there's an incorrect data definition (data not adhering to schema for example) or some other exception.
## Feature
It has been suggested, to allow a more nuanced processing pipe to be possible to define a data quality threshold. If that is breached `SparkJobs` would end with a non-zero exit code too (different from the other exceptions).
## Proposed Solution
Needs to figure out how to specify the data-quality thresholds and then make the checks.
Possibly this can change into an Epic.
コントリビューションガイド
評価
この issue はまだ評価されていません。