AbsaOSS / AbsaOSS/spark-data-standardization
Standardization fails if default value is not defined for a non-nullable array
- 主要语言
- Scala
- 星标
- 7
- 派生
- 2
- 平均合并
- 1 天 18 小时
- 30 天内合并 PR
- 4
描述
## Describe the bug
The error is:
```
Caused by: java.lang.IllegalStateException: No default value defined for data type array
at za.co.absa.standardization.types.CommonTypeDefaults.getDataTypeDefaultValue(CommonTypeDefaults.scala:63) ~[pramen-components_2.12_3.5-0.3.5-SNAPSHOT.jar:0.3.5-SNAPSHOT]
at za.co.absa.standardization.types.CommonTypeDefaults.$anonfun$getDataTypeDefaultValueWithNull$1(CommonTypeDefaults.scala:71) ~[pramen-components_2.12_3.5-0.3.5-SNAPSHOT.jar:0.3.5-SNAPSHOT]
at scala.util.Try$.apply(Try.scala:213) ~[scala-library-2.12.18.jar:?]
at za.co.absa.standardization.types.CommonTypeDefaults.getDataTypeDefaultValueWithNull(CommonTypeDefaults.scala:71) ~[pramen-components_2.12_3.5-0.3.5-SNAPSHOT.jar:0.3.5-SNAPSHOT]
```
But the natural default value for an array is the empty array.
## To Reproduce
1. Create a schema with a non-nullable array column.
2. Try to standardize data with this schema while not specifying a default value for the array column.
3. See the error.
## Expected behavior
Standardization should use empty array as the default value for array type automatically.
## Business Value
--
## Screenshots
--
## Additional context
--
贡献指南
这个仓库没有索引到贡献指南
评估
这个 Issue 还没有评估数据。