Path not filled for root values in Schema upload
- 主要语言
- Scala
- 星标
- 33
- 派生
- 16
- PR 合并指标
- 30 天内没有已合并 PR
描述
## Describe the bug
`path` in `SchemaField` for the root fields is not filled/populated during upload.
It is not validated in any way. Now we have old schemas with this path and new without. So sometimes `getAbsolutePath` works and sometimes it returns duplicates.
Most probably the root of the cause is in `SparkMenasSchemaConvertor#convertSparkToMenasFields`
This also makes trouble for the export/import feature. Currently, we have reached a stable enough point. The fix should come with a change to the SQL database and a full model audit.
### Expected
```scala
private val schemaFieldRoot = SchemaField(
name = "AnyStruct",
`type` = "struct",
path = "AnyStruct",
elementType = None,
containsNull = None,
nullable = true,
metadata = Map.empty,
children = Seq(schemaFieldChildOne, schemaFieldChildTwo)
)
```
### Current
```scala
private val schemaFieldRoot = SchemaField(
name = "AnyStruct",
`type` = "struct",
path = "",
elementType = None,
containsNull = None,
nullable = true,
metadata = Map.empty,
children = Seq(schemaFieldChildOne, schemaFieldChildTwo)
)
```
贡献指南
评估
这个 Issue 还没有评估数据。