SparkJobs return code based on data quality
Abierto
Conformance
feature
priority: low
Standardization
under discussion
- Lenguaje dominante
- Scala
- Estrellas
- 33
- Forks
- 16
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Descripción
## Background
`SparkJobs` return non-zero return code when there's an incorrect data definition (data not adhering to schema for example) or some other exception.
## Feature
It has been suggested, to allow a more nuanced processing pipe to be possible to define a data quality threshold. If that is breached `SparkJobs` would end with a non-zero exit code too (different from the other exceptions).
## Proposed Solution
Needs to figure out how to specify the data-quality thresholds and then make the checks.
Possibly this can change into an Epic.
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.