AbsaOSS / AbsaOSS/enceladus

Path not filled for root values in Schema upload

未关闭
#1,905 0 条评论 1 个 reaction 已指派 0 人 在 GitHub 查看
bug priority: undecided
主要语言
Scala
星标
33
派生
16
PR 合并指标
30 天内没有已合并 PR

描述

## Describe the bug
`path` in `SchemaField` for the root fields is not filled/populated during upload.
It is not validated in any way. Now we have old schemas with this path and new without. So sometimes `getAbsolutePath` works and sometimes it returns duplicates.

Most probably the root of the cause is in `SparkMenasSchemaConvertor#convertSparkToMenasFields`

This also makes trouble for the export/import feature. Currently, we have reached a stable enough point. The fix should come with a change to the SQL database and a full model audit.

### Expected
```scala
private val schemaFieldRoot = SchemaField(
name = "AnyStruct",
`type` = "struct",
path = "AnyStruct",
elementType = None,
containsNull = None,
nullable = true,
metadata = Map.empty,
children = Seq(schemaFieldChildOne, schemaFieldChildTwo)
)
```

### Current
```scala
private val schemaFieldRoot = SchemaField(
name = "AnyStruct",
`type` = "struct",
path = "",
elementType = None,
containsNull = None,
nullable = true,
metadata = Map.empty,
children = Seq(schemaFieldChildOne, schemaFieldChildTwo)
)
```

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。