AbsaOSS / AbsaOSS/enceladus

SparkJobs return code based on data quality

Abierto
#1,542 2 comentarios 0 reacciones 0 asignados Ver en GitHub
Conformance feature priority: low Standardization under discussion
Lenguaje dominante
Scala
Estrellas
33
Forks
16
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

## Background
`SparkJobs` return non-zero return code when there's an incorrect data definition (data not adhering to schema for example) or some other exception.

## Feature
It has been suggested, to allow a more nuanced processing pipe to be possible to define a data quality threshold. If that is breached `SparkJobs` would end with a non-zero exit code too (different from the other exceptions).

## Proposed Solution
Needs to figure out how to specify the data-quality thresholds and then make the checks.
Possibly this can change into an Epic.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.