Field name validation
- Vorherrschende Sprache
- Scala
- Sterne
- 33
- Forks
- 16
- PR-Merge-Kennzahlen
- Keine gemergten PRs in 30 T.
Beschreibung
## Background
If a field name for standardization is not valid for _parquet_ format, the whole standardization is executed and the process fails on saving to _parquet_ file.
## Feature
It would be much more efficient to verify the column names as part of the verification at standardization start and fail early.
## Proposed Solution
Possible places to do the validation at:
* `SchemaValidator` object
* `FieldValidator` class
Beitragsleitfaden
Rechercherichtung
Start by inspecting the `SchemaValidator` object and `FieldValidator` class to determine where standardization-start verification is performed. Reproduce the invalid parquet field-name case, then add validation before standardization proceeds and verify that invalid names fail early rather than only when saving the parquet file.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- scala, spark
- Bereich
- data-engineering
- Issue-Typ
- Feature
- Schwierigkeit
- 3/5
- Geschätzter Aufwand
- 1-2 Tage
- Aktivitätsstatus
- Veraltet
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 42/100