AbsaOSS / AbsaOSS/enceladus

Field name validation

Open
#959 0 comments 0 reactions 0 assignees View on GitHub
feature priority: undecided Standardization under discussion
Dominant language
Scala
Stars
33
Forks
16
PR merge metrics
No merged PRs in 30d

Description

## Background
If a field name for standardization is not valid for _parquet_ format, the whole standardization is executed and the process fails on saving to _parquet_ file.

## Feature
It would be much more efficient to verify the column names as part of the verification at standardization start and fail early.

## Proposed Solution
Possible places to do the validation at:
* `SchemaValidator` object
* `FieldValidator` class

Contributor guide

Open the contributing guide

Research direction

Start by inspecting the `SchemaValidator` object and `FieldValidator` class to determine where standardization-start verification is performed. Reproduce the invalid parquet field-name case, then add validation before standardization proceeds and verify that invalid names fail early rather than only when saving the parquet file.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.