Field name validation
- Dominant language
- Scala
- Stars
- 33
- Forks
- 16
- PR merge metrics
- No merged PRs in 30d
Description
## Background
If a field name for standardization is not valid for _parquet_ format, the whole standardization is executed and the process fails on saving to _parquet_ file.
## Feature
It would be much more efficient to verify the column names as part of the verification at standardization start and fail early.
## Proposed Solution
Possible places to do the validation at:
* `SchemaValidator` object
* `FieldValidator` class
Contributor guide
Research direction
Start by inspecting the `SchemaValidator` object and `FieldValidator` class to determine where standardization-start verification is performed. Reproduce the invalid parquet field-name case, then add validation before standardization proceeds and verify that invalid names fail early rather than only when saving the parquet file.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100