Path not filled for root values in Schema upload
- Dominant language
- Scala
- Stars
- 33
- Forks
- 16
- PR merge metrics
- No merged PRs in 30d
Description
## Describe the bug
`path` in `SchemaField` for the root fields is not filled/populated during upload.
It is not validated in any way. Now we have old schemas with this path and new without. So sometimes `getAbsolutePath` works and sometimes it returns duplicates.
Most probably the root of the cause is in `SparkMenasSchemaConvertor#convertSparkToMenasFields`
This also makes trouble for the export/import feature. Currently, we have reached a stable enough point. The fix should come with a change to the SQL database and a full model audit.
### Expected
```scala
private val schemaFieldRoot = SchemaField(
name = "AnyStruct",
`type` = "struct",
path = "AnyStruct",
elementType = None,
containsNull = None,
nullable = true,
metadata = Map.empty,
children = Seq(schemaFieldChildOne, schemaFieldChildTwo)
)
```
### Current
```scala
private val schemaFieldRoot = SchemaField(
name = "AnyStruct",
`type` = "struct",
path = "",
elementType = None,
containsNull = None,
nullable = true,
metadata = Map.empty,
children = Seq(schemaFieldChildOne, schemaFieldChildTwo)
)
```
Contributor guide
Assessment
This issue has not been assessed yet.