AbsaOSS / AbsaOSS/enceladus

Converting to splittable input will fail if nested column has parquet incompatible name

Open
#960 0 comments 0 reactions 0 assignees View on GitHub
bug priority: undecided Standardization
Dominant language
Scala
Stars
33
Forks
16
PR merge metrics
No merged PRs in 30d

Description

## Describe the bug
If the input file is a unsplittable format, **Standardization** first converts it to a _parquet_ file. At this step it renames the fields per metadata values. but only on top level. After reading the _parquet_ file the fields are renamed back.

There are two issues in this process:
1. Te nested columns still can have _parquet_ invalid field names, that will cause the saving of splittable file to fail
1. The nature of schema definition and _sourcecolumn_ metadata usage does not guarantee that the renaming creates unique field names.

## Expected behaviour
Robust process when converting to splittable format.

## Additional context
Bonus solution - make the splittable input in a more elegant way.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.