Converting to splittable input will fail if nested column has parquet incompatible name
- Lingua principale
- Scala
- Stelle
- 33
- Fork
- 16
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
## Describe the bug
If the input file is a unsplittable format, **Standardization** first converts it to a _parquet_ file. At this step it renames the fields per metadata values. but only on top level. After reading the _parquet_ file the fields are renamed back.
There are two issues in this process:
1. Te nested columns still can have _parquet_ invalid field names, that will cause the saving of splittable file to fail
1. The nature of schema definition and _sourcecolumn_ metadata usage does not guarantee that the renaming creates unique field names.
## Expected behaviour
Robust process when converting to splittable format.
## Additional context
Bonus solution - make the splittable input in a more elegant way.
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.