AbsaOSS / AbsaOSS/spark-commons

Implement error handling by putting the info into _Spark_'s standard error column (`String`)

Abierto
#86 0 comentarios 0 reacciones 0 asignados Ver en GitHub
enhancement
Lenguaje dominante
Scala
Estrellas
7
Forks
0
Merge medio
19 h 57 min
PR fusionados (30 d)
1

Descripción

## Background
One of the provided `ErrorHandling` implementations. Title is actually little misleading, point is to write the errors into string column, and the column name should default into `spark.sql.columnNameOfCorruptRecord` (See [Runtime SQL Configuration](https://spark.apache.org/docs/latest/configuration.html#runtime-sql-configuration))

## Feature
Write errors into a `StringType` column, by converting each error submit filed into a string and concatenating them with a delimiter. The column name should/might default to `spark.sql.columnNameOfCorruptRecord`

## Proposed Solution
Solution Ideas:
* Configurable error delimiter (delimiter between different errors), default `\n`
* Configurable error field delimiter, default `,`
* Enable quoting? Probably yes

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.