Converting to splittable input will fail if nested column has parquet incompatible name
- Ngôn ngữ chính
- Scala
- Star
- 33
- Fork
- 16
- Chỉ số merge pull request
- Không có pull request nào được merge trong 30 ngày
Mô tả
## Describe the bug
If the input file is a unsplittable format, **Standardization** first converts it to a _parquet_ file. At this step it renames the fields per metadata values. but only on top level. After reading the _parquet_ file the fields are renamed back.
There are two issues in this process:
1. Te nested columns still can have _parquet_ invalid field names, that will cause the saving of splittable file to fail
1. The nature of schema definition and _sourcecolumn_ metadata usage does not guarantee that the renaming creates unique field names.
## Expected behaviour
Robust process when converting to splittable format.
## Additional context
Bonus solution - make the splittable input in a more elegant way.
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.