SparkJobs return code based on data quality
Offen
Conformance
feature
priority: low
Standardization
under discussion
- Vorherrschende Sprache
- Scala
- Sterne
- 33
- Forks
- 16
- PR-Merge-Kennzahlen
- Keine gemergten PRs in 30 T.
Beschreibung
## Background
`SparkJobs` return non-zero return code when there's an incorrect data definition (data not adhering to schema for example) or some other exception.
## Feature
It has been suggested, to allow a more nuanced processing pipe to be possible to define a data quality threshold. If that is breached `SparkJobs` would end with a non-zero exit code too (different from the other exceptions).
## Proposed Solution
Needs to figure out how to specify the data-quality thresholds and then make the checks.
Possibly this can change into an Epic.
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.