SparkJobs return code based on data quality
Aperta
Conformance
feature
priority: low
Standardization
under discussion
- Lingua principale
- Scala
- Stelle
- 33
- Fork
- 16
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
## Background
`SparkJobs` return non-zero return code when there's an incorrect data definition (data not adhering to schema for example) or some other exception.
## Feature
It has been suggested, to allow a more nuanced processing pipe to be possible to define a data quality threshold. If that is breached `SparkJobs` would end with a non-zero exit code too (different from the other exceptions).
## Proposed Solution
Needs to figure out how to specify the data-quality thresholds and then make the checks.
Possibly this can change into an Epic.
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.