AbsaOSS / AbsaOSS/spark-commons
Implement error handling by putting the info into _Spark_'s standard error column (`String`)
- 主要语言
- Scala
- 星标
- 7
- 派生
- 0
- 平均合并
- 19 小时 57 分钟
- 30 天内合并 PR
- 1
描述
## Background
One of the provided `ErrorHandling` implementations. Title is actually little misleading, point is to write the errors into string column, and the column name should default into `spark.sql.columnNameOfCorruptRecord` (See [Runtime SQL Configuration](https://spark.apache.org/docs/latest/configuration.html#runtime-sql-configuration))
## Feature
Write errors into a `StringType` column, by converting each error submit filed into a string and concatenating them with a delimiter. The column name should/might default to `spark.sql.columnNameOfCorruptRecord`
## Proposed Solution
Solution Ideas:
* Configurable error delimiter (delimiter between different errors), default `\n`
* Configurable error field delimiter, default `,`
* Enable quoting? Probably yes
贡献指南
这个仓库没有索引到贡献指南
评估
这个 Issue 还没有评估数据。