AbsaOSS / AbsaOSS/enceladus

Path not filled for root values in Schema upload

Ouverte
#1,905 0 commentaires 1 réaction 0 personnes assignées Voir sur GitHub
bug priority: undecided
Langage dominant
Scala
Étoiles
33
Forks
16
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

## Describe the bug
`path` in `SchemaField` for the root fields is not filled/populated during upload.
It is not validated in any way. Now we have old schemas with this path and new without. So sometimes `getAbsolutePath` works and sometimes it returns duplicates.

Most probably the root of the cause is in `SparkMenasSchemaConvertor#convertSparkToMenasFields`

This also makes trouble for the export/import feature. Currently, we have reached a stable enough point. The fix should come with a change to the SQL database and a full model audit.

### Expected
```scala
private val schemaFieldRoot = SchemaField(
name = "AnyStruct",
`type` = "struct",
path = "AnyStruct",
elementType = None,
containsNull = None,
nullable = true,
metadata = Map.empty,
children = Seq(schemaFieldChildOne, schemaFieldChildTwo)
)
```

### Current
```scala
private val schemaFieldRoot = SchemaField(
name = "AnyStruct",
`type` = "struct",
path = "",
elementType = None,
containsNull = None,
nullable = true,
metadata = Map.empty,
children = Seq(schemaFieldChildOne, schemaFieldChildTwo)
)
```

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.