AbsaOSS / AbsaOSS/enceladus

Converting to splittable input will fail if nested column has parquet incompatible name

Aperta
#960 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
bug priority: undecided Standardization
Lingua principale
Scala
Stelle
33
Fork
16
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

## Describe the bug
If the input file is a unsplittable format, **Standardization** first converts it to a _parquet_ file. At this step it renames the fields per metadata values. but only on top level. After reading the _parquet_ file the fields are renamed back.

There are two issues in this process:
1. Te nested columns still can have _parquet_ invalid field names, that will cause the saving of splittable file to fail
1. The nature of schema definition and _sourcecolumn_ metadata usage does not guarantee that the renaming creates unique field names.

## Expected behaviour
Robust process when converting to splittable format.

## Additional context
Bonus solution - make the splittable input in a more elegant way.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.