AbsaOSS / AbsaOSS/spark-commons

Implement error handling by putting the info into _Spark_'s standard error column (`String`)

未關閉
#86 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
enhancement
主要語言
Scala
星號
7
分支
0
平均合併
19 小時 57 分鐘
30 天內合併 PR
1

描述

## Background
One of the provided `ErrorHandling` implementations. Title is actually little misleading, point is to write the errors into string column, and the column name should default into `spark.sql.columnNameOfCorruptRecord` (See [Runtime SQL Configuration](https://spark.apache.org/docs/latest/configuration.html#runtime-sql-configuration))

## Feature
Write errors into a `StringType` column, by converting each error submit filed into a string and concatenating them with a delimiter. The column name should/might default to `spark.sql.columnNameOfCorruptRecord`

## Proposed Solution
Solution Ideas:
* Configurable error delimiter (delimiter between different errors), default `\n`
* Configurable error field delimiter, default `,`
* Enable quoting? Probably yes

貢獻指南

這個儲存庫沒有索引到貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。