Converting to splittable input will fail if nested column has parquet incompatible name
- Dominant language
- Scala
- Stars
- 33
- Forks
- 16
- PR merge metrics
- No merged PRs in 30d
Description
## Describe the bug
If the input file is a unsplittable format, **Standardization** first converts it to a _parquet_ file. At this step it renames the fields per metadata values. but only on top level. After reading the _parquet_ file the fields are renamed back.
There are two issues in this process:
1. Te nested columns still can have _parquet_ invalid field names, that will cause the saving of splittable file to fail
1. The nature of schema definition and _sourcecolumn_ metadata usage does not guarantee that the renaming creates unique field names.
## Expected behaviour
Robust process when converting to splittable format.
## Additional context
Bonus solution - make the splittable input in a more elegant way.
Contributor guide
Assessment
This issue has not been assessed yet.