AbsaOSS / AbsaOSS/enceladus

SparkJobs return code based on data quality

Aperta
#1,542 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Conformance feature priority: low Standardization under discussion
Lingua principale
Scala
Stelle
33
Fork
16
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

## Background
`SparkJobs` return non-zero return code when there's an incorrect data definition (data not adhering to schema for example) or some other exception.

## Feature
It has been suggested, to allow a more nuanced processing pipe to be possible to define a data quality threshold. If that is breached `SparkJobs` would end with a non-zero exit code too (different from the other exceptions).

## Proposed Solution
Needs to figure out how to specify the data-quality thresholds and then make the checks.
Possibly this can change into an Epic.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.