SparkJobs return code based on data quality
Ouverte
Conformance
feature
priority: low
Standardization
under discussion
- Langage dominant
- Scala
- Étoiles
- 33
- Forks
- 16
- Métriques de merge des PR
- Aucune PR mergée en 30 j
Description
## Background
`SparkJobs` return non-zero return code when there's an incorrect data definition (data not adhering to schema for example) or some other exception.
## Feature
It has been suggested, to allow a more nuanced processing pipe to be possible to define a data quality threshold. If that is breached `SparkJobs` would end with a non-zero exit code too (different from the other exceptions).
## Proposed Solution
Needs to figure out how to specify the data-quality thresholds and then make the checks.
Possibly this can change into an Epic.
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.