Converting to splittable input will fail if nested column has parquet incompatible name
- Vorherrschende Sprache
- Scala
- Sterne
- 33
- Forks
- 16
- PR-Merge-Kennzahlen
- Keine gemergten PRs in 30 T.
Beschreibung
## Describe the bug
If the input file is a unsplittable format, **Standardization** first converts it to a _parquet_ file. At this step it renames the fields per metadata values. but only on top level. After reading the _parquet_ file the fields are renamed back.
There are two issues in this process:
1. Te nested columns still can have _parquet_ invalid field names, that will cause the saving of splittable file to fail
1. The nature of schema definition and _sourcecolumn_ metadata usage does not guarantee that the renaming creates unique field names.
## Expected behaviour
Robust process when converting to splittable format.
## Additional context
Bonus solution - make the splittable input in a more elegant way.
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.