AbsaOSS / AbsaOSS/enceladus

Converting to splittable input will fail if nested column has parquet incompatible name

Offen
#960 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
bug priority: undecided Standardization
Vorherrschende Sprache
Scala
Sterne
33
Forks
16
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

## Describe the bug
If the input file is a unsplittable format, **Standardization** first converts it to a _parquet_ file. At this step it renames the fields per metadata values. but only on top level. After reading the _parquet_ file the fields are renamed back.

There are two issues in this process:
1. Te nested columns still can have _parquet_ invalid field names, that will cause the saving of splittable file to fail
1. The nature of schema definition and _sourcecolumn_ metadata usage does not guarantee that the renaming creates unique field names.

## Expected behaviour
Robust process when converting to splittable format.

## Additional context
Bonus solution - make the splittable input in a more elegant way.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.